Are Nash Equilibriums Killing Agile Initiatives?
Over the past two years I have seen a lot of debate about the success of Agile software development. Agile methodologies can produce great results. This is well documented. Yet, in many companies, they don't.
This has lead many people to question Agile. Some reject it altogether. However, the root cause of the problem isn't in the Agile methodologies. The root cause that makes Agile fail is in the companies adopting Agile methodologies.
Let's look back about ten years, when Agile was beginning to gather momentum. It was believed that if a company was functionally organized, like this:
Introducing an Agile software development method in part of the organization like this:
would cause everyone in the organization to reevaluate their strategies, so that the organization could reorganize into a flow organization, like this:
It turned out not to be that simple. An important reason is that most business organizations are designed to be in a stable state. All major players know that if they change their strategy, and the other major players don't, they will loose. It looks a bit like this:
This is a simple example with three players: A, B, and C. If A goes Agile, A can win only if B and C reorganize their part of the organization too. If either B or C chooses not to reorganize, A will loose.
Suppose, for example, that A charges its internal customer B per hour for software development services. By introducing Agile, A might be able to develop software 5 times faster than before, and would thus make 80% less money than before, even though the company as a whole would benefit from reduced lead times and improved product quality. A can benefit from Agile only if A, B and C together change the internal economic system of the company. That is unlikely to happen.
It is worse than that. B and C might have to reorganize to the extent that they cease to exist in order for the company to reap the benefits of Agile. For example, a company may no longer need a Project Management Department, because though Agile teams have leaders, they don't necessarily have project managers. Of course, the Project Management Department will oppose such changes.
Game theorists have a name for such situations: Nash equilibriums. Wikipedia describes a Nash Equilibrium like this:
This is the crux of the matter: Successfully introducing Agile requires a paradigm shift for the entire organization. Most organizations are not prepared for that. Few Agile initiatives have top management support allowing them to change parts of the organization outside the software development department.
That is not to say change is impossible. It is not. It is merely very difficult. On the up side, the difficulties are to a large extent inherent in the organizational structure of most companies. Once a company goes Agile successfully, it will have a more adaptable organizational structure, which makes it the organization easier to change in the future.
This has lead many people to question Agile. Some reject it altogether. However, the root cause of the problem isn't in the Agile methodologies. The root cause that makes Agile fail is in the companies adopting Agile methodologies.
Let's look back about ten years, when Agile was beginning to gather momentum. It was believed that if a company was functionally organized, like this:
Suppose, for example, that A charges its internal customer B per hour for software development services. By introducing Agile, A might be able to develop software 5 times faster than before, and would thus make 80% less money than before, even though the company as a whole would benefit from reduced lead times and improved product quality. A can benefit from Agile only if A, B and C together change the internal economic system of the company. That is unlikely to happen.
It is worse than that. B and C might have to reorganize to the extent that they cease to exist in order for the company to reap the benefits of Agile. For example, a company may no longer need a Project Management Department, because though Agile teams have leaders, they don't necessarily have project managers. Of course, the Project Management Department will oppose such changes.
Game theorists have a name for such situations: Nash equilibriums. Wikipedia describes a Nash Equilibrium like this:
If each player has chosen a strategy and no player can benefit by changing his or her strategy while the other players keep theirs unchanged, then the current set of strategy choices and the corresponding payoffs constitute a Nash equilibrium.A system may have many possible Nash equilibriums. There is no guarantee that a Nash equilibrium is optimal for the system as a whole. Most are not. However, it is often very difficult to move from one Nash equilibrium to another. To do it successfully, all players must be made aware that a better state is attainable and they must trust each other to change.
This is the crux of the matter: Successfully introducing Agile requires a paradigm shift for the entire organization. Most organizations are not prepared for that. Few Agile initiatives have top management support allowing them to change parts of the organization outside the software development department.
That is not to say change is impossible. It is not. It is merely very difficult. On the up side, the difficulties are to a large extent inherent in the organizational structure of most companies. Once a company goes Agile successfully, it will have a more adaptable organizational structure, which makes it the organization easier to change in the future.
Comments
When I've "introduced Agile" I've done it for the sake of creating success, not for the sake of "being Agile." To me, success is at the level I have influence over (individual, associates, team, department) and must be considered in the context of the organization. For example, I've introduced selected Agile practices on a team in order to make that team more successful in the (non-Agile) organization.
This has worked well for me; of course, it also means I haven't changed any "entire organizations" to be Agile.
--mj
And I can certainly see how the lack of embracing any change across the entire organization makes it very difficult to reap the benefits of that change for a small subset of that organization.
That being said I have used agile methods on teams that were largely not embraced our job. We still got a lot of out of it, we just had to set up our firewall appropriately.
Things like TDD, iterative development, and people driven proceses like retrospectives and daily standups can help keep projects from going off the rails even if the entire organization hasn't fully embraced agile.
agile methodology isn't about pace of development. It is about being able to react to changed requirements and about reducing the risk of unclear/miscommunicated requirements.
The interdependency problem arises if dependent teams only partially shift to agile development methods.
Agile methodology depends on almost-instant feedback and communication. If the "customer" (or product owner) of a team cannot provide that feedback in a timely manner (in this setting meaning that the "downstream" team cannot integrate and evaluate the artifacts delivered to them), agile methods fail.
There is no innate need to change a whole organisation to "agile".
BUT, an organization that cannot adopt Agile, probably cannot adopt other things that it needs either. The organization is quite likely to die, and sooner rather than later. (The average lifespan of a business organization is 12.5 years according to Arie De Geus, http://2020Foresight.blogspot.com/2003_04_16_2020foresight_archive.html )
What I recommend, is going Agile using a Systems Thinking perspective, not an Agile perspective. (The shortest description I have written of how to do it is 400 pages long, so I won't repeat it here.)
If you want to make developing software more fun and efficient, just go. Interleave study and practice. Study a bit, practice a bit, repeat...
If you want full scale Agile adoption, i.e. something your organization as a whole will benefit from, I recommend a bit of reading before going ahead:
John Kotter's Heart of Change provides a good template for organizational change. http://bit.ly/66pXs3
Bill Dettmer's The Logical Thinking Process provides you with the tools for planning the change. http://bit.ly/5AIvLp (Scroll down the page to watch my video review or visit YouTube for a slightly larger version http://www.youtube.com/watch?v=UQXXCclDakE )
Chet Richards Certain to Win provides the strategic insights you need. http://bit.ly/6YtC8B
Here are two basic insights that may serve you:
People want to survive on their own terms.
Strategy is a game of interaction and isolation. The first principle of strategy is to focus your forces where they will do the most good. (Hint: The software development department is usually not the strongest power center in the organization.)
Yes we will. The process is speeded up by the recession. Over the past few months I have seen several business articles proclaiming that small business is the next big business.
However, that is under the assumption that the organizational paradigm remains the same: functionally organized and hierarchical.
There are ways for organizations to remain agile regardless of size. Companies that grow and remain agile include The Virgin Group and W.L. Gore Ltd. There are many others. What they have in common are common values throughout the organization, common goal, decentralized decision making, flow organization and network structure.
I believe the next generation of business people may catch on. The current generation, fortunately with with delightful individual exceptions, is too set in its ways.
My dictionary agrees with you. Thanks! not the only spelling mistake in the article. I'll fix them.
I agree. Agile can do a lot of good even if the rest of the organization fails to adapt.
I do not want to discourage anyone from trying. I just want to point out some of the difficulties involved. Agilists are not the first who try to improve business organizations: TQM, Six sigma, business reengineering, TOC, Lean, etc. All have met with limited success overall, even if there are companies who have been spectacularly successful.
However, each failure provides us with information we can use to increase the chance of success the next time around. We just have to decide to be truly Agile and learn.
Thank you! I am trying to spark a bit of debate, so I am delighted we disagree.
I will need a blog post to express a counter argument though, so please bear with me until I can find the time to write one. No more than a day or two, I hope.
There is a very simply way to make agile succeed. It's called a management meeting. Department heads must choose to take on agile roles and be certain that they can live with it. This is best accomplished by organization goal setting and by working out a workable definitive path that each department can commit to.
Without that, in some departments, a method for "becoming agile ready" is simply not known and the cost is anticipated to be too great.
For instance, you can never make development more agile than your "Q/A department" (if you have one) chooses to be. If your Q/A demands that your product release is 100% ready before they see any of, you can never change. That may be their only choice, and other things may block them from changing how they do business.
On the other hand, your development group need to adopt enough agile methods - like scrum in a whole-hearted way. Just putting daily problems "up on a board," making your developers stand up during meetings is not agile.
In short, your company needs to make the decision to become agile - or not. Saying that other departments "won't go along" does not mean all of you should make this decision as a set of random events. That too is entropy. Don't blame Mr. Nash for what your management can or can't do.
Entropy: information: "(communication theory) a numerical measure of the uncertainty of an outcome; "
http://wordnetweb.princeton.edu/perl/webwn?s=entropy
Organizations tend to increase entropy over time, true. It is not quite the same thing I am discussing here. (I have built an "increasing entropy" model in my book Tempo!. I hope I will get it published eventually.)
The Nash equilibria model explains organizational inertia. The organization may be functioning well in a steady state, but is unable to change when circumstances dictate it should. It is not the only such model, but it is a useful one. (There are several such models. Some are generic, but you can also build models for specific companies that yield greater insight into specific situations. That is one of the things I do for a living.)
I am not blaming Mr. Nash, I am grateful to him because he came up with a useful model for understanding what is happening.
You are quite right that management meetings can help resolve the situation. It is simple in principle, but usually difficult in execution.
One of the most prevalent strategies managers use in functional organizations is keeping secrets from other managers. You do not change that with a single meeting. You may need what amounts to several years of group therapy with a skilled facilitator. (Check Argyris below for examples.)
Openly disclosing strategies in order to find a way to move to an improved state is part of how you get out of a Nash equilibrium. However, getting people to accept and understand paradigm shifts is difficult. (On a neuroscience level, they actually have to grow new structures in their brains. http://bit.ly/54Q5kx )
Read Chris Argyris's Knowledge for Action, to get an idea about the work involved. http://bit.ly/8G44eY
It can be done. Alan Mulally did it at Ford. (Lost the link, unfortunately)
Roland deVries did it when he united the South African Army. http://bit.ly/6DmCXo
but it was not easy things to do. We should not forget that we are asking people to change habits they have had all their life. That is difficult to do even if one wants to and no external force is constraining you. Think of giving up smoking or radically changing political and religious beliefs. Can be done, but not easy.
thanks, I enjoyed it.