This has lead many people to question Agile. Some reject it altogether. However, the root cause of the problem isn't in the Agile methodologies. The root cause that makes Agile fail is in the companies adopting Agile methodologies.
Let's look back about ten years, when Agile was beginning to gather momentum. It was believed that if a company was functionally organized, like this:
This is a simple example with three players: A, B, and C. If A goes Agile, A can win only if B and C reorganize their part of the organization too. If either B or C chooses not to reorganize, A will loose.
Suppose, for example, that A charges its internal customer B per hour for software development services. By introducing Agile, A might be able to develop software 5 times faster than before, and would thus make 80% less money than before, even though the company as a whole would benefit from reduced lead times and improved product quality. A can benefit from Agile only if A, B and C together change the internal economic system of the company. That is unlikely to happen.
It is worse than that. B and C might have to reorganize to the extent that they cease to exist in order for the company to reap the benefits of Agile. For example, a company may no longer need a Project Management Department, because though Agile teams have leaders, they don't necessarily have project managers. Of course, the Project Management Department will oppose such changes.
Game theorists have a name for such situations: Nash equilibriums. Wikipedia describes a Nash Equilibrium like this:
If each player has chosen a strategy and no player can benefit by changing his or her strategy while the other players keep theirs unchanged, then the current set of strategy choices and the corresponding payoffs constitute a Nash equilibrium.A system may have many possible Nash equilibriums. There is no guarantee that a Nash equilibrium is optimal for the system as a whole. Most are not. However, it is often very difficult to move from one Nash equilibrium to another. To do it successfully, all players must be made aware that a better state is attainable and they must trust each other to change.
This is the crux of the matter: Successfully introducing Agile requires a paradigm shift for the entire organization. Most organizations are not prepared for that. Few Agile initiatives have top management support allowing them to change parts of the organization outside the software development department.
That is not to say change is impossible. It is not. It is merely very difficult. On the up side, the difficulties are to a large extent inherent in the organizational structure of most companies. Once a company goes Agile successfully, it will have a more adaptable organizational structure, which makes it the organization easier to change in the future.