Re-imagining Agile part 1: Why it shouldn't be done, and how to do it
There is a long running thread at LinkedIn, well over 200 posts now, about the principles of Agile software development. The issue is whether the principles can be improved, whether they should be improved, and how that might be done. I have written about the thread before, but there are still a couple of things mulling about in my head.
I got into a real writing fit this time, so I have split the article into parts. This is the first. More will follow.
I wrote in the thread that, technically speaking, I do believe the Agile principles can be improved upon. I also wrote that I believe it is a bad idea to try:
On the other hand, re-imagining Agile is an interesting exercise. I like to tinker with systems, so let's do it for the fun of it, just to see where we end up.
If I were to design a framework for developing hyper-effective software development methods, where would I start?
I am a systems thinker, so I want to design from the outside in. What is the outermost layer of interest? The paradigm! Our paradigms, our worldviews, shape the systems we build. Paradigms are the tools we use to make sense of the world around us.
(If you are an Agile toolhead and like it that way, you can save yourself some trouble by not reading the rest of this. You will be confused, upset, and lose faith in what you are currently doing. Trust me, I went through it myself many years ago.)
Why should we begin with paradigms? There are three important reasons:
Fig 1: Sources of the new paradigm
This is a simplified figure of course. It covers only literary sources, omits many important works, and does not show the relationships between them. Nevertheless, it does serve as a high level map of the ideas shaping the basic worldview I intend to use.
This is important: You can check the map! You can read up on the ideas and test their validity yourself! Thus, you can find flaws, or alternatives that suit you better, and improve on it. (Please tell me when you do - helps me improve my own paradigms.) There is nothing stopping you from developing many different maps, and pick the one most appropriate to the situation at hand. Technically, it is a paradigm transcendent meta-framework. I call it Strategic Navigation, because it was to a large intent inspired by Bill Dettmer's book Strategic Navigation.
Scientific Management, the currently dominant paradigm, asks you to take its ideas on faith. It claims to be the best, regardless of circumstances. And, despite the name, it isn't very scientific at all. Scientific Management used to be more scientific than it is today, but unfortunately, the initial curiosity that begat Scientific Management solidified into dogma very quickly. (Unfortunately, the same thing is happening with Agile today. This death of curiosity is often called "maturity".)
For brevity's sake, I will assume that you know the basics even if you haven't read the same sources. You need to know a little bit about Systems Thinking, TPS/Lean, Theory Of Constraints (TOC), Maneuver Conflict and Theory X/Y. I'll introduce you to some of it in this article, but do not expect a softly cushioned introduction. I'd need to write another book for that, or perhaps make a movie.
Given the paradigm in Figure 1, we can determine that the system we want to design Agile for consists of the following entities:
I will deliberately refrain from describing the system in more detail than this just yet. We will look at a couple of different system designs later on.
That is of course true, but it does raise a few questions:
Nor is growth a measure of success. Organizations grow in order to achieve their purpose better, and to enhance their chance of survival, but growth for its own sake is cancer, not success.
It is difficult for an enterprise to be successful in the long term if it is pre-destined to have a short life-span.
These models often go into details of the characteristics of each stage, but they rarely provide insight into how this cycle can be prolonged, or even stopped entirely.
When you look a bit closer at the life-span of organizations, some things stand out:
Given the paradigm in Figure 1, this is not surprising. While an organization can croak for a variety of reasons, we can easily explain why hierarchical Command & Control organizations have a poor survival record.
Fig 2: Hierarchical structures are prone to information overload
Because important decisions are taken at high levels in the organization, managers need to deal with more and more information the higher in the structure they are. This causes serious information overload, which gets progressively worse the larger the organization is.
Fig 3: Organizational complexity leads to paralysis
As Figure 3 shows, when the organization tries to defend itself against the information overload caused by its own growth, it actually makes the problem worse. Eventually, it grows so paralyzed it cannot adapt to changing external conditions, and kills itself off.
There are more problems related to information overload. For example, managers try to defend themselves against information overload by dealing with aggregated information. However, when you aggregate information, important signals will disappear in the random noise generated from other sources. For example, you can have a project portfolio that does very well even if one of the projects in it is in deep trouble. A manager that looks only at portfolio information will miss this.
A very serious problem pertains to how managers try to control their organizations in functional C&C hierarchies. Look at Figure 4. It shows various types of measures managers can take in order to change an organization.
Fig 4: Leverage points for intervening in an organization
Most managers don't even know this range of options exist. They focus on setting targets, the weakest intervention point. The result is bad management.
Fig 5: Managing by setting targets is bad management
You might wonder why setting targets is bad. Well, the assumption behind setting targets is that the people responsible for achieving the target are also the people who are in control of the results. This is not true. There are many factors, controlled by many different entities, that influence the results.
First of all, there are factors outside the organization itself. For example, you can deliver the same product or service to two different customers who have the same problem, and one customer will love you, the other will hate you, because the customers have different expectations. (Remember the brouhaha over the iPhone 4 antenna trouble just a short while ago. To most people it isn't a problem at all, to others it was the end of the world, or at least a sign of the impending doom of Apple.)
Of course, an organization that outsources a lot of work becomes more vulnerable to variation. This is rarely taken into account when outsourcing deals are made, but it is important to the performance of the organization outsourcing the work.
Second, there are internal factors. If the organization is a functional hierarchy there are many such factors (because of all the functional sub-divisions) and they are often in different control structures (different legs in the organization chart).
Fig 6: Most causes of results are outside the control of the responsible party
You get something like figure 6. You may be responsible for achieving a target, but you have very limited control over the factors influencing the results.
As a result, you get into a completely ridiculous situation. Mapping target levels to process behavior using a Process Behavior Chart, you get something like figure 7.
Fig 7: Targets drive dumb behavior no matter how you set them.
To make it worse, reward and punishment systems are often set up so that they are triggered by random variation in results, not by how well people have actually done the work. Random rewards and punishments are a great way to drive people a bit nutty, and make them do crazy things.
Add to this that scientific research shows extrinsic rewards, like money, has a tendency to short-circuit the brain. The scientific community has known about this since the sixties. Thanks to modern neuroscience we know a bit about why this happens: Extrinsic rewards trigger the pleasure center in the brain and deactivates the center responsible for socially well adjusted behavior. Pretty much the same effect cocaine has. (Read up on it: Drive by Daniel Pink, Punished by Rewards by Alfie Kohn.)
I won't write much about performance evaluations here, but you might want to check out this link. Suffice it to say, most performance evaluation systems in use today tend to do more harm than good to the companies that use them.
If you have read this far, you might start to think people are terminally stupid. Not true! However, people adapt to the systems they are part of. If the system is built based on the assumption that people are selfish and stupid, then you will get selfish and stupid behavior.
Theory X, formulated by Douglas McGregor, is a theory of human motivation that encapsulates the views of Taylor and many, many, more people. As it turns out, people who have Theory X mindsets will design organizations a certain way. See Figure 8.
Fig 8: Theory X affects how organizations are designed.
When you have an organization based on Theory X, the members of that organization will fear being punished for failure. However, only failures of commission (you do something and fail) are punished. Failures of omission (you didn't do something you ought to have done) are usually not punished. The result is that you get very risk averse organizations. People in them will consistently miss opportunities, and fail to correct systemic problems. (Edwards Deming estimated 94% of all problems in businesses to be systemic. I believe his estimate was too low.)
People will also be tightly controlled, which makes life excruciatingly painful. Eventually the pain goes away, but only because people get numb and institutionalized.
Theory X is self-fulfilling: Scare people and control their every action, and they will lose their motivation, they will behave stupidly, and the only thing motivating them to work will be the pay-check.
At this point you might wonder: What about the Successful Large Enterprise? How can a company grow to giant size and make tons of money if it is a wreck by design and run by zombies?
First of all, if you look at the situation right now, most companies are built on the same ideas and operated in the same manner. Therefore they perform about equally well.
Second, if you look at how business organizations have developed over the past 150 years or so, they could have developed very differently. For example, Henry Ford had two different plants operated in slightly different ways. He used one of the plants as a template for how to develop manufacturing at Ford. The plants were visited by a group of Japanese engineers and business people in the late forties. They were more interested in how the other plant worked, went back home, and built Toyota. (The full story is in Lean Cost Management: Accounting for Lean by Establishing Flow by James Huntzinger.)
The way Ford chose to operate was a success for customers and shareholders, but not for the people working in his factories:
After the financial crisis in 2008 about 10,000 people lost their jobs in the region of Sweden were I live. Many of those who lost their jobs did so quite unnecessarily. To put it bluntly, it is easier to fire people than it is to figure out how to be profitable in a recession. Still, for many companies it is quite doable to move forward in a recession because the competition slows down and miss even more opportunities than usual. I contacted a representative of the union I am a member of, and asked if they were interested in trying to prevent some of this. I proposed that the union should collaborate with management in order to figure out how to minimize the damage from the recession, or even take advantage of it.
I shouldn't have bothered. The response was what you would expect:
"We don't do that! We are not interested in doing anything until after our members are laid off."
This is exactly the dysfunctional behavior John Seddon writes about.
Sometimes I wonder what would have happened if Henry Ford had been more interested in that other plant.
What if that train crash had not happened? What if people had realized that a system designed to prevent low probability high impact events like train wrecks, are not very good for running normal high probability, low impact business operations? What if people had realized that a system designed to prevent one kind of problem often causes other kinds of problems?
It is not always the best system that wins in this type of environment: The QWERTY keyboard is designed to slow typists down (because early typewriter mechanisms could not keep up with the speed of typists), VHS beat Betamax, and a crackpot system for business organization and management beat better systems.
The phenomenon is known as path dependency. It explains how past events affect the future. For example, Java was designed to appeal to C++ programmers. C++ was designed to appeal to C programmers. Thus, the design of C and C++ have influenced Java. Java has had a lot of impact on the design of C#. That does not mean Java and C# are the best language designs possible. People use it because other people use it.
It is the same thing with management paradigms. Theory X reinforces itself, so once it became dominant, the ball just kept rolling. It does not mean Theory X is a correct description of human nature. Nor does it make Theory X suitable as a foundation for building business organization.
All it means is that shit happens. (Path dependency is thoroughly explained in Business Dynamics by John Sterman.)
The ideas I have written about here, Systems Thinking, variation, etc., are ideas that prompted Agile. (If you don't recognize them, I suggest you read a few books by Kent Beck, Jim Highsmith, Allistair Cockburn and other people who created Agile. Pay attention to the reference lists at the end of the books.)
There is another idea that is fundamental to the development of Agile. It is called Theory Y. Theory Y was developed by Douglas McGregor, who also formulated Theory X. Like Theory X, Theory Y is self-fulfilling (it becomes true if you act as if it is true), and it is self-reinforcing. However, the basic assumptions about human nature are the opposite of Theory X. If you believe in Theory Y, you will develop organizational structures and management methods that are completely different from the structures in Theory X companies. Figure 9 shows Theory Y
Fig 9: Theory Y - The foundation of Agile and the next generation of business organizations
Note how the job of managers change. Their job is now to create the right conditions for people to enjoy and excel at their work. To do this, they need to understand how to use, and not to use, all the levers shown in Figure 4.
It is a completely different way to manage, and you know what: A lot of people are to set in their ways to make the change! If you are a lord in a feudal society, a shift to democracy would rob you of your position and power. In a similar way, if you are a manager in a Theory X organization, you are quite likely to lose position and power if the organization transforms to Theory Y.
In some Theory Y organizations, like Ricardo Semler's Semco and W. L. Gore & Associates, members (they don't have employees in the traditional sense), elect their own bosses. (Terri Kelly, the CEO of W. L. Gore & Associates was elected to her position.) How many bosses do you know that would be elected to their position, if the people voting were the people working for them?
There are Theory Y organizations that are not quite so democratized, for example Richard Branson's Virgin Group. Still, it is a bit of a stretch imagining a C-level executive from GM doing very well at Virgin. GM managers and Virgin managers make people laugh, but for entirely different reasons.
Agile is designed to work in Theory Y organizations. It is designed to enable Theory Y organizations to vastly outperform Theory X organizations!
Theory X and Theory Y are fundamentally incompatible. Going back to the LinkedIn thread for a moment: The change that must be made make Agile more easily adopted by large enterprise organizations is to replace Theory Y with Theory X. Of course, if you do that, Agile won't be Agile anymore. We'll be back to old waterfall methods, or worse.
Like the early evangelists I believed the proof would be in the pudding: Agile produced results, great results, so of course management would be delighted.
In retrospect, a pretty naive idea. Agile was, and is, a misfit with most companies. For awhile, that was difficult to accept. Given the current system, managers trying to adopt Agile methods take great risks. Agile methods tend to expose the problems proliferating in other parts of the company, so a manager supporting Agile can easily make a lot of enemies. (This is the same problem managers have when they try to adopt Lean, TOC, and other systems that can improve the performance of their companies.)
However, I have changed my own perspective a bit: Agile won't be successful by being adopted by dysfunctional dinosaurs. Agile will be successful by being adopted by the new breed of companies that will replace them.
Agile's big mistake was designing from the inside out: trying to improve a part of a system that just isn't designed for that kind of improvement.
So, if we want to improve Agile, we need to design from the outside in. We need to improve the fit with the next generation of business organizations. What will that next generation look like?
Watch out for Part 2!
I got into a real writing fit this time, so I have split the article into parts. This is the first. More will follow.
I wrote in the thread that, technically speaking, I do believe the Agile principles can be improved upon. I also wrote that I believe it is a bad idea to try:
- I do not know anyone who has the mandate to change the basic principles of Agile. The original signatories might, but I doubt they want to.
- There is an existing body of work based on the current version of the principles. That body of work will not be redesigned just because someone releases The Principles of Agile 2.0. Scrum won't change, Extreme Programming won't change, the enterprise organizations that keep misunderstanding Agile won't change.
On the other hand, re-imagining Agile is an interesting exercise. I like to tinker with systems, so let's do it for the fun of it, just to see where we end up.
If I were to design a framework for developing hyper-effective software development methods, where would I start?
I am a systems thinker, so I want to design from the outside in. What is the outermost layer of interest? The paradigm! Our paradigms, our worldviews, shape the systems we build. Paradigms are the tools we use to make sense of the world around us.
(If you are an Agile toolhead and like it that way, you can save yourself some trouble by not reading the rest of this. You will be confused, upset, and lose faith in what you are currently doing. Trust me, I went through it myself many years ago.)
Why should we begin with paradigms? There are three important reasons:
- All paradigms are models, and all models have limitations. If we understand the models we use, we can consciously select the thought model best suited to deal with a particular problem.
- If we understand our own paradigms we are in a position to detect when they fail us. This is a prerequisite for improving them, or switching to better ones.
- The currently dominant paradigm is seriously out of touch with reality. I'll write more about this.
Fig 1: Sources of the new paradigm
This is a simplified figure of course. It covers only literary sources, omits many important works, and does not show the relationships between them. Nevertheless, it does serve as a high level map of the ideas shaping the basic worldview I intend to use.
This is important: You can check the map! You can read up on the ideas and test their validity yourself! Thus, you can find flaws, or alternatives that suit you better, and improve on it. (Please tell me when you do - helps me improve my own paradigms.) There is nothing stopping you from developing many different maps, and pick the one most appropriate to the situation at hand. Technically, it is a paradigm transcendent meta-framework. I call it Strategic Navigation, because it was to a large intent inspired by Bill Dettmer's book Strategic Navigation.
Scientific Management, the currently dominant paradigm, asks you to take its ideas on faith. It claims to be the best, regardless of circumstances. And, despite the name, it isn't very scientific at all. Scientific Management used to be more scientific than it is today, but unfortunately, the initial curiosity that begat Scientific Management solidified into dogma very quickly. (Unfortunately, the same thing is happening with Agile today. This death of curiosity is often called "maturity".)
For brevity's sake, I will assume that you know the basics even if you haven't read the same sources. You need to know a little bit about Systems Thinking, TPS/Lean, Theory Of Constraints (TOC), Maneuver Conflict and Theory X/Y. I'll introduce you to some of it in this article, but do not expect a softly cushioned introduction. I'd need to write another book for that, or perhaps make a movie.
Given the paradigm in Figure 1, we can determine that the system we want to design Agile for consists of the following entities:
- Customers - This is usually a group of many different customers. Quite often, it is possible to divide this entity in several different sub-types with different properties.
- The business organization
- The subsystem in the business organization responsible for delivering value to a customer or type of customers.
I will deliberately refrain from describing the system in more detail than this just yet. We will look at a couple of different system designs later on.
Why do we need new ideas?
We don't need new ideas unless there is something wrong with the ones we already have. In the LinkedIn thread that inspired this article I suggested that it is better to redesign the enterprises to fit reality, than to redesign Agile to fit dysfunctional enterprises. The response was that if you have a Successful Large Enterprise, there is no need to change it.That is of course true, but it does raise a few questions:
- Successful on whose terms? The owner's? (All of the owner's? Which owner groups?) The people who are members of the organization? (All of them? Only particular groups?) The customers of the organization? Depending on the situation, and the paradigms of the people involved, you may be able to satisfy all (win-win, perspective of bounty), or only a few, perhaps only one (win-lose, perspective of scarcity).
- What is success? Making as much money as possible? Hardly. Money is a tool. Having more money means "having a more powerful tool". If you have a lot of money but don't know what to do with it, you are pitiful, not successful.
Nor is growth a measure of success. Organizations grow in order to achieve their purpose better, and to enhance their chance of survival, but growth for its own sake is cancer, not success.
Death to the successful - How successful growth can turn into corporate cancer
Most business organizations do not survive long, even if they grow very large. The life-span for a large organization is about the same as for a human. While the average life-span of humans is growing longer, the average life-span of business organizations isn't.It is difficult for an enterprise to be successful in the long term if it is pre-destined to have a short life-span.
There are plenty of life-cycle models that show the life-cycle of organizations. It usually goes something like this:
- Start-up
- Growth
- Maturity
- Decline
- Death (or, with a few lucky ones, Reinvention)
These models often go into details of the characteristics of each stage, but they rarely provide insight into how this cycle can be prolonged, or even stopped entirely.
When you look a bit closer at the life-span of organizations, some things stand out:
- Some organizations last a lot longer than the norm. (Religious organizations, countries...)
- The short-lived organizations are functional hierarchies with Command & Control cultures
- The long-lived organizations have network structures, are purposeful organizations, or combine both features
Given the paradigm in Figure 1, this is not surprising. While an organization can croak for a variety of reasons, we can easily explain why hierarchical Command & Control organizations have a poor survival record.
This illustration shows how information flows in a hierarchical structure:
Fig 2: Hierarchical structures are prone to information overload
Because important decisions are taken at high levels in the organization, managers need to deal with more and more information the higher in the structure they are. This causes serious information overload, which gets progressively worse the larger the organization is.
Fig 3: Organizational complexity leads to paralysis
As Figure 3 shows, when the organization tries to defend itself against the information overload caused by its own growth, it actually makes the problem worse. Eventually, it grows so paralyzed it cannot adapt to changing external conditions, and kills itself off.
There are more problems related to information overload. For example, managers try to defend themselves against information overload by dealing with aggregated information. However, when you aggregate information, important signals will disappear in the random noise generated from other sources. For example, you can have a project portfolio that does very well even if one of the projects in it is in deep trouble. A manager that looks only at portfolio information will miss this.
A very serious problem pertains to how managers try to control their organizations in functional C&C hierarchies. Look at Figure 4. It shows various types of measures managers can take in order to change an organization.
Fig 4: Leverage points for intervening in an organization
Most managers don't even know this range of options exist. They focus on setting targets, the weakest intervention point. The result is bad management.
Fig 5: Managing by setting targets is bad management
You might wonder why setting targets is bad. Well, the assumption behind setting targets is that the people responsible for achieving the target are also the people who are in control of the results. This is not true. There are many factors, controlled by many different entities, that influence the results.
First of all, there are factors outside the organization itself. For example, you can deliver the same product or service to two different customers who have the same problem, and one customer will love you, the other will hate you, because the customers have different expectations. (Remember the brouhaha over the iPhone 4 antenna trouble just a short while ago. To most people it isn't a problem at all, to others it was the end of the world, or at least a sign of the impending doom of Apple.)
Of course, an organization that outsources a lot of work becomes more vulnerable to variation. This is rarely taken into account when outsourcing deals are made, but it is important to the performance of the organization outsourcing the work.
Second, there are internal factors. If the organization is a functional hierarchy there are many such factors (because of all the functional sub-divisions) and they are often in different control structures (different legs in the organization chart).
Fig 6: Most causes of results are outside the control of the responsible party
You get something like figure 6. You may be responsible for achieving a target, but you have very limited control over the factors influencing the results.
As a result, you get into a completely ridiculous situation. Mapping target levels to process behavior using a Process Behavior Chart, you get something like figure 7.
Fig 7: Targets drive dumb behavior no matter how you set them.
To make it worse, reward and punishment systems are often set up so that they are triggered by random variation in results, not by how well people have actually done the work. Random rewards and punishments are a great way to drive people a bit nutty, and make them do crazy things.
Add to this that scientific research shows extrinsic rewards, like money, has a tendency to short-circuit the brain. The scientific community has known about this since the sixties. Thanks to modern neuroscience we know a bit about why this happens: Extrinsic rewards trigger the pleasure center in the brain and deactivates the center responsible for socially well adjusted behavior. Pretty much the same effect cocaine has. (Read up on it: Drive by Daniel Pink, Punished by Rewards by Alfie Kohn.)
I won't write much about performance evaluations here, but you might want to check out this link. Suffice it to say, most performance evaluation systems in use today tend to do more harm than good to the companies that use them.
If you have read this far, you might start to think people are terminally stupid. Not true! However, people adapt to the systems they are part of. If the system is built based on the assumption that people are selfish and stupid, then you will get selfish and stupid behavior.
Theory X - The root of all evil
As I have mentioned, most business organizations are based on Scientific Management. Scientific Management was created by Frederick Taylor. Taylor believed workers are lazy, stupid, and motivated only by money. (Don't take my word for it. Read The Principles of Scientific Management.) He created a management paradigm designed to force lazy, stupid, disinterested people to work.Theory X, formulated by Douglas McGregor, is a theory of human motivation that encapsulates the views of Taylor and many, many, more people. As it turns out, people who have Theory X mindsets will design organizations a certain way. See Figure 8.
Fig 8: Theory X affects how organizations are designed.
When you have an organization based on Theory X, the members of that organization will fear being punished for failure. However, only failures of commission (you do something and fail) are punished. Failures of omission (you didn't do something you ought to have done) are usually not punished. The result is that you get very risk averse organizations. People in them will consistently miss opportunities, and fail to correct systemic problems. (Edwards Deming estimated 94% of all problems in businesses to be systemic. I believe his estimate was too low.)
People will also be tightly controlled, which makes life excruciatingly painful. Eventually the pain goes away, but only because people get numb and institutionalized.
Theory X is self-fulfilling: Scare people and control their every action, and they will lose their motivation, they will behave stupidly, and the only thing motivating them to work will be the pay-check.
At this point you might wonder: What about the Successful Large Enterprise? How can a company grow to giant size and make tons of money if it is a wreck by design and run by zombies?
First of all, if you look at the situation right now, most companies are built on the same ideas and operated in the same manner. Therefore they perform about equally well.
Second, if you look at how business organizations have developed over the past 150 years or so, they could have developed very differently. For example, Henry Ford had two different plants operated in slightly different ways. He used one of the plants as a template for how to develop manufacturing at Ford. The plants were visited by a group of Japanese engineers and business people in the late forties. They were more interested in how the other plant worked, went back home, and built Toyota. (The full story is in Lean Cost Management: Accounting for Lean by Establishing Flow by James Huntzinger.)
The way Ford chose to operate was a success for customers and shareholders, but not for the people working in his factories:
Despite higher wages Ford's new system suffered from stupendous labour turnover. Newly hired workers lasted an average of only three months. Many walked off the job without any formal notification, and were presumed to have quit after missing five days of work: the notion of the 'five-day man' was born and accounted for 70 percent of the workers leaving Ford. Mass production systems were and are monotonous, demoralizing places to work. Trade unions grew out of the 20th century systems of mass production; current-management union practices serve to maintain the dysfunctional relationship. The relationship won't change until the system - the way work is designed and managed - changes too.
Systems Thinking in the Public Sector, by John SeddonJohn seddon is right. Here is a personal experience:
After the financial crisis in 2008 about 10,000 people lost their jobs in the region of Sweden were I live. Many of those who lost their jobs did so quite unnecessarily. To put it bluntly, it is easier to fire people than it is to figure out how to be profitable in a recession. Still, for many companies it is quite doable to move forward in a recession because the competition slows down and miss even more opportunities than usual. I contacted a representative of the union I am a member of, and asked if they were interested in trying to prevent some of this. I proposed that the union should collaborate with management in order to figure out how to minimize the damage from the recession, or even take advantage of it.
I shouldn't have bothered. The response was what you would expect:
"We don't do that! We are not interested in doing anything until after our members are laid off."
This is exactly the dysfunctional behavior John Seddon writes about.
Sometimes I wonder what would have happened if Henry Ford had been more interested in that other plant.
Why management often looks like a train wreck
In 1841 there was a train crash in the United States. To prevent similar crashes from occurring a system for controlling operations by dividing responsibilities and authority, and by having a system of reporting and checks. This was the origin of the hierarchical organization chart. (Systems Thinking in the Public Sector, p. 48, by John Seddon)What if that train crash had not happened? What if people had realized that a system designed to prevent low probability high impact events like train wrecks, are not very good for running normal high probability, low impact business operations? What if people had realized that a system designed to prevent one kind of problem often causes other kinds of problems?
The secret of enterprise success: Path dependency
Small events can cause very large effects in a certain type of systems: Systems dominated by reinforcing feedback loops. Expanding markets are dominated by reinforcing feedback loops, and it was in this type of environment the large enterprises we have today were born.It is not always the best system that wins in this type of environment: The QWERTY keyboard is designed to slow typists down (because early typewriter mechanisms could not keep up with the speed of typists), VHS beat Betamax, and a crackpot system for business organization and management beat better systems.
The phenomenon is known as path dependency. It explains how past events affect the future. For example, Java was designed to appeal to C++ programmers. C++ was designed to appeal to C programmers. Thus, the design of C and C++ have influenced Java. Java has had a lot of impact on the design of C#. That does not mean Java and C# are the best language designs possible. People use it because other people use it.
It is the same thing with management paradigms. Theory X reinforces itself, so once it became dominant, the ball just kept rolling. It does not mean Theory X is a correct description of human nature. Nor does it make Theory X suitable as a foundation for building business organization.
All it means is that shit happens. (Path dependency is thoroughly explained in Business Dynamics by John Sterman.)
Good things happen too: Theory Y and Agile
Fortunately, good things happen too. So far, I haven't really covered anything new. I haven't even covered all the bases. For example, Cost Accounting was developed based on ideas stemming from Scientific Management. Cost Accounting is one of the main drivers of poor decision making in organizations. I won't go into the reasons, but you can check it out yourself. The references are in Figure 1.The ideas I have written about here, Systems Thinking, variation, etc., are ideas that prompted Agile. (If you don't recognize them, I suggest you read a few books by Kent Beck, Jim Highsmith, Allistair Cockburn and other people who created Agile. Pay attention to the reference lists at the end of the books.)
There is another idea that is fundamental to the development of Agile. It is called Theory Y. Theory Y was developed by Douglas McGregor, who also formulated Theory X. Like Theory X, Theory Y is self-fulfilling (it becomes true if you act as if it is true), and it is self-reinforcing. However, the basic assumptions about human nature are the opposite of Theory X. If you believe in Theory Y, you will develop organizational structures and management methods that are completely different from the structures in Theory X companies. Figure 9 shows Theory Y
Fig 9: Theory Y - The foundation of Agile and the next generation of business organizations
Note how the job of managers change. Their job is now to create the right conditions for people to enjoy and excel at their work. To do this, they need to understand how to use, and not to use, all the levers shown in Figure 4.
It is a completely different way to manage, and you know what: A lot of people are to set in their ways to make the change! If you are a lord in a feudal society, a shift to democracy would rob you of your position and power. In a similar way, if you are a manager in a Theory X organization, you are quite likely to lose position and power if the organization transforms to Theory Y.
In some Theory Y organizations, like Ricardo Semler's Semco and W. L. Gore & Associates, members (they don't have employees in the traditional sense), elect their own bosses. (Terri Kelly, the CEO of W. L. Gore & Associates was elected to her position.) How many bosses do you know that would be elected to their position, if the people voting were the people working for them?
There are Theory Y organizations that are not quite so democratized, for example Richard Branson's Virgin Group. Still, it is a bit of a stretch imagining a C-level executive from GM doing very well at Virgin. GM managers and Virgin managers make people laugh, but for entirely different reasons.
Agile is designed to work in Theory Y organizations. It is designed to enable Theory Y organizations to vastly outperform Theory X organizations!
Theory X and Theory Y are fundamentally incompatible. Going back to the LinkedIn thread for a moment: The change that must be made make Agile more easily adopted by large enterprise organizations is to replace Theory Y with Theory X. Of course, if you do that, Agile won't be Agile anymore. We'll be back to old waterfall methods, or worse.
Agile's big mistake
When I became interested in Agile in 2002, I became enamored by the possibilities. Like early evangelists, I believed Agile would be like an asphalt flower: The roots would spread, cracking the asphalt, opening ways to transform companies in all areas: organizational structure, management practices, strategy, economic models, sales and marketing. Even back then, I dimly grasped that for a major change in software development methodology to work, other parts of the organization must change too.Like the early evangelists I believed the proof would be in the pudding: Agile produced results, great results, so of course management would be delighted.
In retrospect, a pretty naive idea. Agile was, and is, a misfit with most companies. For awhile, that was difficult to accept. Given the current system, managers trying to adopt Agile methods take great risks. Agile methods tend to expose the problems proliferating in other parts of the company, so a manager supporting Agile can easily make a lot of enemies. (This is the same problem managers have when they try to adopt Lean, TOC, and other systems that can improve the performance of their companies.)
However, I have changed my own perspective a bit: Agile won't be successful by being adopted by dysfunctional dinosaurs. Agile will be successful by being adopted by the new breed of companies that will replace them.
Agile's big mistake was designing from the inside out: trying to improve a part of a system that just isn't designed for that kind of improvement.
So, if we want to improve Agile, we need to design from the outside in. We need to improve the fit with the next generation of business organizations. What will that next generation look like?
Watch out for Part 2!
Comments
Other: "Agile sounds good, but it wouldn't work here. You haven't met my developers."
Me: "What about them?"
Other: "They aren't motivated. They take no initiatives. I could never just help them set a goal and then leave them to it for a couple of weeks. Nothing would happen."
Me: "How do you know that? Have you tried it?"
Other: "No."
Me: "So maybe we should give it a shot and see what happens?"
Other: "Hmm. Maybe"
Of course our expectations on others affect them. How could they not? Thanks again for the article. Keep it up.