Dangerous Myths and Misconceptions about Agile Software Development

I’ve been working with Agile software development since 2001-2002. Over the years, I’ve seen Agile change, a lot!

There has been some genuine development. For example, the rediscovery of Monte Carlo simulation counts, even though ordinary project management has had it since at least 1963. Reference Class Forecasting, aging analysis, user story mapping, Cost of Delay economics, and economic profiles, are all useful innovations we have had after the Agile Manifesto was published. While these things were not necessarily invented in the Agile space, agilists began using them after the manifesto was published.

However, there was a lot more innovation the ten years before the manifesto, than we have seen in the twenty-five years after. Also, Agile is drowning in misconceptions and misinformation. This article is an attempt to correct some of the more egregious misconceptions I have seen. Along the way, I’ll point to some good sources, for those of you inclined to do a bit of fact-checking. (Which, really, should be everyone involved with Agile, or software development in general, or management, politics, or, you know, life…)

In this post, I’ll tackle the following myths and misconceptions:

• The “Agile is a Mindset” delusion
• The Agile Manifesto contains all you need to know about Agile
• Agile = Scrum
• Agile can be installed in an organization, like software is installed on a computer
• Estimates work (you just have to try a bit harder…)
• Story points will improve your estimates
• The Fibonacci series will improve your estimates

I do hope you enjoy the read, and above all else, find it useful.

The “Agile is a Mindset” Delusion

Oh no, dear, sweet summer child! Agile is not a mindset, at least not in the way most people who claim it believe! Agile is a quite extensive body of knowledge, amassed over a period of more than thirty years, and building on even older bodies of knowledge.

If you want to actually do something with Agile, it is much more useful to think of Agile as a set of methodologies, frameworks, and toolkits. The accompanying mindset is something you will acquire by using those methodologies, frameworks, and toolkits.

Originally, Agile methodologies, also called Lightweight Methodologies referred to the methodologies, and toolkits, represented by seventeen people at a meeting at the Snowbird Ski Resort, February 11-13, 2001. Check out the manifesto author biographies at the Agile Manifesto website to find out what bodies of knowledge the authors represented.

If you look at the history of Agile, methodologies like Crystal, Scrum, Adaptive Software Development, Dynamic Systems Development Method, Feature-Driven Development, and eXtreme Programming, and toolkits like Pragmatic Programming, were developed to solve practical problems in the 1990’s. The Agile movement was originally a craftsmanship movement, much more interested in building skill sets than having a particular mindset.

This means Agile practices preceded the Agile Manifesto by nearly a decade! Check out the history page on the Agile Manifesto website if you want to fact-check.

The Agile Manifesto for Software Development extracted a common belief system from those methodologies. It is important to acknowledge that the skills came before the beliefs, otherwise, it is easy to delude oneself into believing the beliefs will work without the skills. You may even lose track of what the actual skills are.

The “Agile is a mindset” misconception has spread because it is deceptively attractive. If Agile is only a mindset, and the mindset is expressed by the Agile Manifesto, then, the reasoning goes, you don’t have to learn anything, you just have to profess your allegiance to the principles in the manifesto.

The Manifesto Contains All You Need to Know

Closely related to the “Agile is a Mindset” delusion, is the idea that the Agile Manifesto itself contains everything you need to know to build Agile teams and organizations.

Unfortunately, a manifesto is not a manual for how to do things. A manifesto is a public declaration of beliefs and intentions. It does not tell you how to do anything. The idea that you can have the mindset without the skill set has lead to many Agile implementations today being little more than cargo cult management.

Let’s test the assumption that the manifesto itself contains what you need to know to build Agile teams and organizations. We can do this by looking at an Agile principle, as expressed in the Agile Manifesto, and check what skills, if any, are needed to put the principle into practice. Here is one Agile principle:

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

What the principle refers to, is practices that make it easy to change the program code, so that changes are cheap and easy to implement.

eXtreme Programming posits that software engineering practices can be used to reduce the cost of change over the product lifecycle.

The idea is from eXtreme Programming. It simply says that good software engineering practices can reduce the cost of change over the entire product lifecycle, and, that if we keep the cost of change down, we can accept changes in requirements late in a project.

It is worth noting that Scrum does not do that! There are no software engineering practices in Scrum. (Originally, Scrum was intended to be used in conjunction with software engineering practices. This has been de-emphasized, because it made Scrum more difficult to sell.)

What software engineering practices help us keep the cost of change low? Let’s have a look at what Robert C. Martin, “Uncle Bob”, has written about it. Robert Martin is the guy who came up with the idea of writing the Agile Manifesto. Martin has written more than a dozen books on software development. The most famous one is Clean Code. In that book, he promotes, among a lot of other things:

• Test-Driven Design (TDD)
• Refactoring
• SOLID, an acronym for five useful design principles
• Design patterns
• The DRY principle (Don’t Repeat Yourself)
• The Law of Demeter, an object-oriented design principle
• Paying attention to code smells and using heuristics. He lists 66 of them.

The book contains a lot more, and as I mentioned, Martin has written more than a dozen of them, all about software engineering.

Don’t forget, Martin is only one voice in the original Agile community! There were many other Agile Manifesto authors who wrote books on different ways of accomplishing the same thing.

Of course, there were also many, many more people who wrote excellent (and sometimes very bad) books about Agile software development in the years following the publishing of the manifesto.

It is pretty clear you need a knowledge of software engineering that is both broad and deep, in order to welcome changing requirements late in the development process.

It’s the same with the other eleven principles: They all require both broad and deep skills, in various areas, not just software engineering, in order to make practical use of them.

For example:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

This one might require you to reorganize not just your own organization, but also your customer’s organization. You might need high levels of skill in Cost of Delay economics, contract negotiation, organizational change, strategy, organizational politics, complexity theory, systems thinking, statistics and variation, epistemology, psychology, neuroscience, queuing theory, and process theory, to pull it off.

Sadly, most organizations don’t have that. It is much easier to reject the idea that they need to learn a lot of stuff, than to actually learn, so reject the idea of continuous learning is what they do.

On the upside, if you are willing to go through the trouble of learning a lot of stuff, you have a great competitive advantage.

Agile=Scrum

No, Agile and Scrum are not the same thing!

Scrum is a lightweight framework for managing projects. It has few rules, but those few rules are immutable. Consequently, the rules of Scrum are considered context independent. (This post is about Agile, not Scrum, so I won’t go into detail about why that is a horrendously bad idea.)

Originally, Scrum was developed for teams with extremely high technical competence that had problems delivering software because of churn (priorities changed faster than work could be implemented) because of stakeholder interference. Scrum solved the problem by collecting all requirements in the Product Backlog, and forcing all stakeholder communication to go via the Product Owner.

Scrum history has been exposed to a lot of revisionist writing. If you want to know the original pupose of Scrum, what it was really designed for, I suggest you find read some of the original Scrum papers and books. I recommend Agile Software Development with Scrum by Ken Schwaber and Mike Beedle.

Agile is an entirely different beast. In the early 90’s, a new generation of software development methodologies evolved. These included Dynamic Systems Development Method, the Crystal family of methodologies, Adaptive Software Development, Scrum, eXtreme Programming, and XBreed.

The proponents of these methods began communicating, and around 1997, a new movement emerged. The common name for the methodologies were Lightweight Methodologies, which distinguished them from older, Heavyweight Methodologies, like the Rational Unified Process (RUP), various Critical Path based methodologies, and Waterfall methodologies like PROPS.

The term Agile Software Development was coined at the aforementioned Snowbird Ski Resort meeting. Nobody was happy with the term “lightweight”. Alistair Cockburn, the creator of the Crystal family of methodologies, was the one who voiced the concerns in the group. This led to discussions, and eventually, the term “Agile” emerged.

The Agile Alliance defines Agile software development like this:

Agile is an umbrella term for a set of frameworks and practices based on the values and principles expressed in the Agile Manifesto and the 12 Principles behind it.

So, Agile is definitely not the same thing as Scrum!

Agile Can be Installed, Like Software

Many change projects fail because management acts as if organizational change is a simple problem, when the reality is that organizational change has both complicated and complex parts.

The projects were the Agile methodologies emerged had great success with them. That created a strong incentive for organizations to “go Agile”. Unfortunately, many organizations misunderstand the very nature of organizational change, and consequently go about it in ways that have a very low probability of success.

About 50% of all Agile transformations fail. Of course, the rate of failure depends a lot on how you define success, but still, even if we do not accept the failure rate as a given, it is clear organizational change is failure prone. Agile is not the only kind of change that is difficult to implement. Depending on which study you look at, Lean has a failure rate between 50% and 98%.

If you look at the reasons for failure cited in research about Agile and Lean transformation, they have a lot in common. Some of the most common causes of failure cited in the research are:

• Lack of top management commitment and involvement
• Lack of communication
• Lack of training and education
• Limited resources

An important observation about the Toyota Production System (TPS), the original, and incredibly successful, version of Lean, is that it was neither invented, nor installed as a complete package. Instead, it evolved! Step by step, Taiichi Ohno, and others, solved problems specific to Toyota, one by one, over an extended period of time. The process began in the 1940’s, and is continuing today.

The Agile methodologies, toolkits, and frameworks evolved the same way. As has been described earlier, Scrum evolved to prevent stakeholder interference. eXtreme Programming evolved during the 80’s and 90’s to reduce risk in a highly volatile environment. The Crystal family of methods evolved as a family in order to fit a wide range of different types of projects at IBM, and so on.

In each case, even if there was considerable overlap in the problems the methodologies were designed to solve, the solutions were different, because the context was different, and all of them evolved over a period of time.

The problem is that in many organizations, management does not want to hear that they need to personally engage in the organization evolving its own ways of working, adapted to solve problems specific to the organization, in ways that fit the organizational context.

Management wants it to be a simple problem, and so managers often choose to act as if it is a simple problem. This can easily cause mismatches between what the organization does, for example hiring a couple of coaches to get all development teams to use Scrum, and the problems the organization has, which may not even have to do with the development teams at all. Getting development teams to use Scrum does not help much if:

• Projects are contractually obligated to deliver once per year.
• The “teams” are functionally organized workgroups.
• The teams lack software engineering skills such as design patterns, refactoring, Test Driven Design, Domain-Driven Design, and SOLID. (This will be even more true if the teams use AI to produce code.)
• Organizational structures prevent close collaboration with real users.
• Teams have external dependencies with long cycle times.
• Developers can’t collaborate because they are physically separated.
• Contracts are fixed price.
• All requirements have been defined up front before the development team even sees them.
• Collaborating Scrum teams have separate product backlogs.
• Team topology is developed ad hoc.
• Teams cannot change direction quickly because of a yearly budget system.
• Teams cannot change direction quickly because of a yearly strategy cycle.
• The development process is sub-optimized because of cost accounting.

…and so on.

It is much better to start where you are, figure out what steps you can take to make things better, implement, and repeat. Big changes are sometimes necessary, like switching from Cost Accounting to Cost of Delay, but in general, small, evolutionary changes, guided by cheap, multiple, parallel experiments, are best.

If you treat Agile as a very large body of knowledge you can draw ideas and inspiration from, that has a better chance of working than trying to install a framework regardless of context.

Estimates Will Work (You Just Have to Try Harder…)

Most estimation methods are based on the idea that the durations you are estimating are normally distributed, and that the averages of the estimates be the same as the averages of the actual durations.

The estimation myth is is both pernicious and incredibly persistent. Estimates frequently fail, but it is usually easier, more convenient, to blame the people making the estimates, than it is to question whether it was a good idea to ask for estimates in the first place.

Most estimation methods make use of the Central Limit Theorem. The theorem states that given the appropriate conditions, the mean of a sample of data will be closer and closer to the mean of the overall population as the sample size increases.

If this holds true of both the real population of job durations, and estimates of job durations, they will both converge towards a mean. If we can calibrate our estimates, the estimates will converge towards the same mean as the real population of job durations.

There is one big problem though:

Software development projects do not have normal distributions of job durations and costs!

Software development projects, and jobs, like User Stories, tasks, Features, and Epics, all have fat tail distributions of duration and cost.

Fat tail distribution of User Story cycle times from a real project.

The figure above shows User Story cycle times from a real project that uses Scrum. The cycle times are not normally distributed, and trying to estimate them is futile.

Scatter plot showing correlation between estimates and real cycle times in a real project.

It is easy to check whether estimates work or not: You can create a scatter plot, like the one above, comparing estimates and actual cycle times. If the estimates work, the data points will form a, slightly wobbly, straight line.

As you can see, there is no such line! The data points are randomly distributed.

As a further check, I calculated the statistical correlation coefficient. This is easy to do in a spreadsheet. As you can see the correlation coefficient is very close to zero, which means there is no meaningful correlation between estimates and cycle times.

In theory, you could have a team that is so technically excellent that it can cut off the long tail of the distribution, so that you do have a normal distribution. In that case, estimates could work. However, this would also require that the team is truly autonomous, with very few external dependencies.

Such teams are very, very rare.

Story Points will Improve Your Estimates

I have already shown you that estimates are highly unlikely to work, so it should come as no surprise that Story Points are just a piece of ritual magic, about as useful as a horoscope. (For the record, horoscopes don’t work!)

Story points did serve a useful purpose once, a long time ago. They were invented by Ron Jeffries, and were first used in eXtreme Programming.

In eXtreme Programming, developers estimated the duration of User Stories in Ideal Days, that is, days where nothing would go wrong, and the developers were left to themselves, so they could work. The Ideal Days were multiplied with a Load Factor to get an estimate of how long it would take to implement the User Story.

For example, if a team has a Load Factor of 3, and a story is estimated to take 2 Ideal Days, the actual estimate would be:

2 Ideal Days x 3 = 6 Work Days

The problem was that stakeholders could not distinguish between an Ideal Day, and an actual work day, and got confused about the time estimates. Jeffries came up with the idea of estimating in points instead. Thus you got:

2 Story Points x 3 = 6 Work Days

The predictive value of Story Points is as good as the value of other kinds of time estimates, which, as I showed in a previous section, is zero.

Ron Jeffries has written an apology for the invention of Story Points. I suggest you read it. Martin Fowler has also written about how Story Points were intended to be used.

Over time, the original purpose, and the mechanism for creating them, was lost, and replaced by folklore, like “Story Points are an estimate of complexity, not duration”.

The important point here is that regardless of how you use them, Story Points don’t work, because estimates don’t work!

The Fibonacci Series Will Improve Your Estimates

Using the Fibonacci series when estimating is another piece of ritual magic we have inherited from eXtreme Programming. (Just to be clear, eXtreme Programming is dear to my heart! It was where my journey with Agile began, and I still consider it to be a great basis for a highly competent team to work. I am not blind to the fact that our knowledge of how software development projects work, has evolved since the 1990’s though.)

The idea is that spacing out estimates according to the Fibonacci series, i.e. 1, 2, 3, 5, 8, 13, 21…, would make the estimates closer to actual durations.

The problem with that, is:

• Using Fibonacci won’t ensure that estimates match the fat tail distribution of real job durations.
• Different teams have very different job duration distributions, so no single estimation trick can magically fit all of them.

Once more: Your team’s estimates almost certainly do not work! Use a scatter plot and calculate the statistical correlation coefficient to check.

That’s all, for now! When I prepared for writing this article, I found several more common misconceptions about Agile software development. I may, or may not, follow this article up with another one, depending on how I decide to spend the rest of my vacation.

Be seeing you!