Re-Imagining Agile part 2: Designing from the outside in

A business that makes nothing but money is a poor business.

– Henry Ford

In the first part of the Re-Imagining Agile series I wrote about some of the ideas Agile is based on. The most basic one is a theory about human behavior and motivation, Theory Y.

I wrote that most business organizations are designed based on a different set of assumptions, Theory X. Moving to Agile therefore represents a paradigm shift, a change in how we think.

Fig 10: The paradigm shift.

Figure 10 illustrates the shift in paradigms. As you can see, the shift is much more extensive than switching from Theory X to Theory Y. You might wish to compare this figure with the reading map in Figure 1 (in Part 1 of this series).

While it is difficult to make the transition, it is quite possible. One organization that has made this shift is the U.S. Marine Corps. They did something really sneaky: They kept the name Command & Control, but they changed the definition of the word control!

...decentralized control works from the bottom up. Command is the exercise of authority and guidance, and control is felt as feedback about the effects of the action taken because thinking is required at all levels.

Tactics, MDCP 1-3 - The U.S. Marine Corps manual on Tactics

It is my belief that what has enabled the Marines to make the paradigm shift was that they had pretty strong Theory Y leanings even before.

For now, I am going to leave the question of how to make an organization, even a large organization, make the basic paradigm shift. Instead, I will focus on what we want to achieve. As Dee Hock, who created VISA, put it, we must imagine what ought to be!

Decisions - the Power of the OODA Loop

We began by defining the paradigm, the system of thought, we need to re-imagine Agile. What is the next layer? We know that we want to build a system based on Theory Y. We know we want to design from the perspective of the customer, but it is not quite that easy. A business organization must make some pretty complicated decisions, like who the customers are, and what particular needs the company should satisfy, the nature of their relationship with customers, how to organize to utilize their resources as well as possible...

Decisions, decisions! Everyone in a business organization must make a lot of decisions, every day. What if we could have a common framework for all decisions throughout the organization? What if the CEO, a software developer, an accountant, and sales person, everyone in the company, including the people who empty the waste baskets, all had a common framework for making decisions?

The purpose of such a framework would be to improve the quality of decisions throughout the organization. Thus, the framework must make it easy to make good decisions, and make them quickly.

As it turns out, there is such a framework. It was developed by U.S Air Force Colonel John Boyd. Originally it was used to describe the decision processes of fighter pilots, but it soon gained traction as a generic model for decision making. The model is called the OODA Loop, and it is at the core of John Boyd's strategy meta-framework, Maneuver Conflict.

Fig. 11: The OODA Loop - An effective decision model

Figure 11 reveals the OODA Loop in all its gory glory. When I first saw it I thought, "What the #%&@ is that?" so don't be taken aback if it looks a bit complex at first.

I will not go into details about how to use the OODA Loop, but I will mention that an important facet is the ability to change speed and direction very quickly. In other words, to be Agile!

Few companies have strategy cycles shorter than a year. Using the Strategic Navigation framework, a business implementation of Maneuver Conflict, strategic cycles for large companies can be as short as a week.

Just like Agile methodologies allow development teams to use short iterations and short delivery cycles, so can business strategists compress strategy cycles with Strategic Navigation.

Just to illustrate the advantages fast strategy cycles can bring:

One company that is very good at compressing their strategy cycles is Apple. First they launched the iPod, then they used the popularity of the iPod to launch iTunes. iTunes, with the addition of the AppStore, became major incentive for customers to buy the iPhone. iTunes and the AppStore, with the addition of the new Bookstore greatly enhance the attractiveness of the iPad. Concurrently, they use Mac computers to increase the value of iPod/iPhone/iPad/iTunes, and iPod/iPhone/iPad/iTunes to increase the value of Mac computers. For example, the video capabilities of iPhone 4 fits neatly with the iMovie program included with all Macs. Most of the competition is far behind, trying to copy the iPhone/iTunes combination, while Apple continuously shapes and reshapes the market.

Since we have a spiffy new paradigm at our disposal, let's use it. We switch from Maneuver Conflict, where we found the OODA Loop, to Queueing Theory, where we find Little's Law:

t = I/T

Where:

t is lead time
I is the number of items in a process
T is the rate at which items are completed

As you probably know, Little's Law is fundamental to Agile software development. It is a natural law, and if Little's Law didn't work, Agile wouldn't work either. (Suggested reading: Lean Software Development and Implementing Lean Software Development by Tom and Mary Poppendieck; Product Development Flow by Donald Reinertsen.)

When I became interested in Maneuver Conflict, and saw the OODA Loop , something went BANG!!! in my head, and the lights turned on.

I suddenly realized that Little's Law is vitally important to the OODA Loop too. The arrows in the OODA Loop diagram show how information flows when we make decisions. The Post-It notes on a kanban board also show how information flows. They are both governed by the same basic law, Little's Law!

We need to make good decisions fast. Little's Law gives us an idea about how to accomplish that: We can either reduce I, or increase T. How do we do that?

We can reduce I in top levels of an organization by making as many decisions as possible at lower levels of the organization. At the same time we engage more people in thinking, and thus increase T.

Quick switch to our Systems Thinking hat: When we decentralize decision-making we shorten information feedback loops, which shortens delay in the loops, which increases the effect and reliability of the feedback. If you look at Figure 4 in Part 1, you can see that we can improve two high impact leverage points, the information flow structure, and the system structure, in one fell swoop.

Looks like Maneuver Conflict, Queueing Theory, and Systems Thinking all lead to the same conclusion. Great!

Here is one implication: If the CEO of your company understands the OODA Loop, and you, a software developer, understand the OODA Loop, you know have a common frame of reference.

Because everyone in the organization can use the OODA Loop as a frame of reference for how they make decisions, everyone in the organization has a shared frame of reference for understanding each other.

Once again, when I write "the OODA Loop", I do not mean just the diagram. I mean the whole package that goes with it.

We now have a starting point for developing an organization designed to take full advantage not just of of Agile software development, but also of modern ideas about how to organize, modern economic ideas, modern ideas about product development...

Figure 9 (in Part 1) showed some of the implications of Theory Y:

• Workers must be encouraged to communicate vertically and laterally. (No organizational silos.)
• There is no need for elaborate hierarchies of control and authority
• Workers should have a say in decisions that affect them.
• Failures that result in learning should be rewarded. (No fear culture! For brevity's sake, I will not go into this point any further.)

We can use the OODA Loop, Little's Law, and the list above to begin designing an agile organization.

We need to design the organization so that it consists of small, autonomous units. That way, the organization can grow without overloading the OODA Loops of top management. At the same time, the organization can quickly reconfigure itself when it needs to do so.

We need to weigh economy of scale against economy of speed. When we are the slightest bit uncertain, we'll choose speed. Speed is very important when the environment changes quickly. And we want to do more than to be able to respond quickly, we want to actually shape the environment. That requires the ability to go fast, and to change speed and direction very quickly for the whole organization!

Judging from our sample companies in Figure 1, we want to keep the main organizational units no larger than 100-200 people. Beyond that, we will loose to much agility. Of course, this is a very general guideline. For example, if your company is an industrial company, you may need to have some very large main units, like a large factory. Still, the principle holds, and even within a large factory, we can strive to make value streams interfere with each other as little as possible. (This is what Lean does with takt time. When variability in demand is high, as in software development, it can be better to separate the value streams as much as possible.)

What would this look like? Well, there is no absolutely fixed structure. You can't have a fixed organization chart, because the structure keeps changing. (I am not joking. For example, Semco, one of the model organizations in Figure 1, does not have organization charts for precisely this reason.) But you can show the basic principle.

Fig 12: Simplified network structure

Figure 12 shows the principle. Keep two things in mind: The figure shows a snapshot in time, and it is highly simplified.

We are designing from the outside in, so let's begin at the point where the business organization has contact with the customer. I am going to assume that the organization builds custom software of some kind. The particulars are different for other kinds of organizations, but the principles of designing from the outside in and designing against demand, i.e. from the need of the customer, remain the same.

I am also going to push the principles of Theory Y, pull and self-organization to the max here. Remember, we are designing what ought to be, not trying to make minor improvements to what already is. Agile is not faster waterfall, so why limit ourselves when we are designing an organization ideal for taking advantage of Agile?

Teams are built around value streams

The modern industrial organization is a vast complex of interdependent relationships, up, down, across, and even "diagonally." In fact, the system is so complex that only collaborative team efforts can make the system work effectively. It is probable that one day we shall begin to draw organization charts as a series of linked groups rather than as a hierarchical structure of individual "reporting" relationships.

The Human Side of Enterprise by Douglas McGregor

Let's assume initial contact is between the customer and a sales-person. The sales-person would be trained in a sales method compatible with Agile software development, for example the Prime Process. The Prime Process has roots in Systems Thinking, and just as important, it is an explicitly ethical sales method with focus on delivering business value to the customer. (See Mastering the Complex Sale by Jeff Thull.)

If you are an Agilist, you and a Prime Process sales person do have a lot in common: shared goals, and shared assumptions about how to achieve them.

The sales person posts the status of negotiations with the customer. Skills needed for a sales team will also be publicly posted. This will most likely include people with design skill and technical skills. People interested in the job can sign up to work on the sales team.

Because people sign up, instead of having someone assign them, we reduce management work. We also eliminate the need for costly systems to track which people are available. Sales people must know whether people are available to do work, so the company will probably have a kanban-like system to ensure that sales people don't over-commit.

Signing up might be as easy as moving a kanban card representing you, the developer, from a resource pool area on a whiteboard, to a project team area on a different board. Put the boards in the cafeteria, and everyone will be able to track what everyone else is doing at all times.

If the sales team does not make the sale, everyone just moves their cards back to the pool. If a sale is made, the team continues to work with the customer. The sales person may remove himself from the team in order to continue selling, but some of the people will remain, which provides continuity for the customer, and for the team itself. If more diverse skills, or more manpower, is needed, a message about the open positions are posted on the board.

You might ask what happens if people don't want to sign up. That, I am afraid, is a Theory X question. Ask instead: Given that people want to work, and want to do good work, what could prevent them from signing up? If you come up with a real obstacle, fix it. If something is preventing you from fixing it, fix that first!

What if there isn't enough people, or anyone with the right skills, available within the company? Post the job openings to a computerized version of the board that is available to the entire business organization. The team can also contact other companies in the same organization directly, or through Power Groups. (A bit more about those later.)

Note that the team does not necessarily have to bring people from other parts of the organization in. If it is advantageous to do so, the team can temporarily move itself to another company for the duration of the project. Profit sharing schemes for handling this are pre-arranged, so no complicated negotiations between business units are necessary.

By this time you probably think I am off my rocker, but there are real organizations that exhibit precisely this kind of flexibility. A couple of examples:

A U.S. Marine Corps team can decide to move from one chain of command to another if it makes it easier for the team to accomplish its mission.

I am a member of Business Network International (BNI), a business referral network. BNI has about 125,000 members. I belong to a team named Gothia Towers. Today, I had lunch with two BNI members from a different team. They invited me to visit their BNI team next Friday, when they have their weekly business meeting. I accepted. In effect, we took a lateral contact, and even temporarily rearranged the team structures.

If you had suggested to the team members who invited me that they should check with the team management first, they would look at you as if you were a bit funny in the head. They would be right. Their team leaders would be justifiably annoyed if they did. (So I am very glad the thought never crossed your mind.)

If the right people cannot be found within the organization, the team can decide to hire people temporarily. They can do this because:

• They have an economic model to guide them (See Product Development Flow by Donald Reinertsen)
• They are not constrained by a pre-determined budget (See Beyond Budgeting by Jeremy hope and Robin Fraser)
• They are self-organizing

A team pulls resources if and when it needs them. If the team grows too large, it restructures itself into sub-teams.

Fig 13: Linked team structure and communications pathways

As figure 13 shows, each person with leadership responsibilities is a member of two teams. This is different from managers in a hierarchical organization having meetings together. The leaders in Figure 13 work together, ensure that the teams they lead collaborate effectively, and make trade-offs where necessary.

As I have already mentioned, anyone in a team is free to contact any other team without going through a vertical hierarchy. There is a strong emphasis on lateral communication.

A third thing that is different, and once again borrowed from the U.S. Marine Corps, is the idea of the Directed Telescope, or pull communications.

In a hierarchical structure, information is usually pushed upwards, through standardized reporting procedures. A network structure will have similar procedures for pushing information inwards. (Though the type of information, and the amount of information, is very different because the paradigm governing what is considered important information is different.)

There is another way to get information: Pulling it. That means someone at the center actively goes out and looks at the situation in order to understand it. This is sometimes known as Management-by-Walking-Around. Lean folks call it genchi genbutsu, “actual place, actual thing”, to go and observe in order to fully grasp a situation. ("Grokking", if you have read Robert Heinlein's Stranger In A Strange Land.)

One reason why pull communications are under-used in hierarchical organizations is that managers are already swamped by information pushed at them. In the network structure we are designing, we do not push nearly as much information to managers. We also present information in different ways, for example by using process behavior charts or TLTP diagrams. This makes it easier for centrally located managers to pull information when needed.

Note that the structure in Figure 13 allows both independent action and close collaboration. The same linked team structure is used throughout the entire organization.

Depending on what they do, some teams last only a short time, other teams may be very long-lived. For example, a team responsible for the grocery section in a supermarket would never disband, even though membership would change over time. (Remember, we are talking about a general structure for business organizations, not just a structure for software development teams.)

In some business organizations, like Semco and W.L. Gore & Associates, team leaders are elected by their teams. It might be a good idea to require elected leaders to take advanced training in the leadership skills mandated by the basic paradigm. Of course, everyone in the organization should have basic training, and plenty of opportunity to learn on their own if they so choose. It might look like Figure 14.

Fig 14: Mandatory training of elected leaders

Companies

For the purpose of the model, I have assumed that the organization is fairly large, and that the basic business unit is an independent company. A structure like this provides economic protection. If one company fails, other companies will be affected as little as possible.

It is also possible to have other companies absorb people from the failed companies, or use the freed manpower to start new businesses. This is a strategy that the Virgin Group has used successfully for many years. (See Business Stripped Bare, by Sir Richard Branson.)

A company is responsible for one or more value streams. Figure 15 shows the principle.

Fig 15: A company is a linked team structure with the responsibility for managing one or more value streams

I mentioned a size limit on companies. Keeping companies small keeps information loads under control. The exact size limit depends on what business the company is in.

When a company grows to its limit, part of it is split off to form a new company. The Virgin Group employs a strategy where the new company must find a market for itself that is different from the markets of all other companies in the network. That is why Virgin is so diversified: Music, hotels, spaceflight, soft drinks, etc. This prevents different companies in the network from competing with each other.

Traditional enterprises also divide themselves into companies, but they do it in a different manner. It is quite common for an enterprise to start a new company for the purpose of executing one step in a process. The result is a chain of tightly interdependent companies. If one company fails, so does the process, and the other companies in the chain can't get the revenue they need to survive. This makes all companies in the chain very vulnerable. To make it worse, the companies usually use Cost Accounting practices, which means they all strive to be as profitable as possible, which means they have to fight each other.

It is very important to avoid this trap, and set companies up so that they really are built around end-to-end value streams.

Power Groups

As I showed in the section about teams, it is quite likely that these companies have opportunities to collaborate now and then.

Sometimes team level collaboration is not enough. For example, companies might share information on a continuous basis, or work together to execute major operations.

When a company believes it can benefit from formal collaboration with one or more other companies, it can attach itself to a Power Group. Power Groups share information, and may help each other in other ways, as they see fit. Note that a company decides for itself which power group, or groups, it wishes to apply to. The power group then decides whether to accept the application. A power group may of course elect to invite a company.

The Core

There is no "up" in a network organization of this type, and thus no "top management". However, at the center, there is a core, and a core team. The core team is responsible for the system as a whole. Following the same structural pattern as before, most members of the core team will also be leaders of companies, service units, or power groups.

The core team ensures that the organization moves towards the overall system goal. Depending on the organization, they may be the people setting the systemic goal, or not. For example, at the Virgin Group, this is the people ensuring that the brand values are adhered to throughout the organization. Using Maneuver Conflict terminology, they are responsible for the organization's Noble Vision.

The Noble Vision is an idea that unites not only the people within the organization, but also the organization and its customers. Think about Apple brand loyalty, a football team and its fans, and you'll get the basic idea. There is also a moral dimension to the Noble Vision concept, but I will not go into detail here. Suffice to say that the Noble Vision must be something that you and your team mates believe is worth striving for.

One very important service that should be handled by the core in most organizations is basic management training. Every leader in the organization, whether a Scrum Master or the CEO, must have the same basic training. Look at the Orientation step in Figure 11. There must be a shared base here, or it will be impossible to develop the trust and understanding necessary for decentralized command.

A related service that might benefit from centralization is management consulting. For example, the Virgin Group has an internal management consultant organization for the purpose of helping to solve thorny management problems.

A basic rule is to centralize services that benefit from economy of scale, and decentralize services that benefit from economy of speed.

IOHAI

I have mentioned the need for leadership training a couple of times in this article series. When John Boyd developed Maneuver Conflict and the idea of a Noble Vision, he also conceived a leadership model, IOHAI. The IOHAI model is a framework for training leaders that can help move an organization towards a Noble Vision.

Boyd considered IOHAI to be the key to enabling vitality and growth. It is certainly not the only management or leadership framework that can be used, but it is a very useful one. It incorporates ideas of paradigm transcendence and continuous learning.

Fig 16: IOHAI - Theme for vitality and growth

I wrote my book Tempo! so that it can be used as an IOHAI training manual for leaders at all levels in a business organization. Once again, I will not go into details here, but when you dig down into IOHAI, it becomes very practical and action oriented.

Design notes and useful Intermediate Objective Maps

We can summarize our design goal for the organization, and the necessary conditions that must be satisfied in order to achieve the goal. Figure 17 shows what that might look like.

Fig 17: A generic organizational IO Map

Though I did not write about it, I used Self-Determination Theory (SDT) as a guide when developing the organizational structure. Figure 18 shows how, according to SDT, intrinsic motivation can be stimulated.

Fig 18: Self-Determination Theory - The building blocks of intrinsic motivation

As it turns out, most processes have a lot in common. We don't need a template specifically for how to construct software development methodologies. We can construct a generic template for process design, and use that to design every process in the organization. Figure 19 shows what such a template might look like.

Fig 19: An IO Map for process development

Using the same basic ideas for designing all processes is very important. If you have a mismatch in design philosophy, the processes will not match each other very well, which will lead to a lot of waste of effort and time, and reduce the quality of the end result.

Agile principles re-imagined

This article series was inspired about a discussion in a LinkedIn forum about Agile principles. Given what has been done so far, what about the current Agile principles? Will they stand as is, or can they be improved upon? Yes, I believe they can. (As you may recall from part 1, I answered the question "should they be changed?" with "No!". But I answered "can they be changed?" with "Yes!")

Let's begin by defining the word "principle". It was clear from the LinkedIn forum that the word means different things to different people. I will go with the definition in my dictionary:

principle - a fundamental truth or proposition that serves as the foundation for a system of belief or behavior or for a chain of reasoning

There are 12 Agile principles, which I think is a bit much. The principles themselves are not simple. In addition to being too many, several are unnecessarily wordy.

Ironically, principle 10 is:

Simplicity--the art of maximizing the amount of work not done--is essential.

Some of the principles are best described as opinion, plausible on the surface, but unsubstantiated:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

The above sparked a lot of conversation in the LinkedIn thread. It is clear that it leaves many considerations out of the picture, and that face-to-face conversation is not always the best alternative. For example, if the information conveyed is very complex, it may be necessary to write it down in addition to talking about it. In many situations, face-to-face conversation is not possible.

Finally, some things are just wrong:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Why not reflect and fix immediately when a problem crops up? I am for regular introspections, but I believe it should complement immediate stop-and-fix, not replace it. Lean companies have andon and kaizen, not one or the other.

Here is my attempt at Re-imagining the Agile principles:

• Study and apply the paradigm!
• Be curious!
• Be tenacious!
• Design from the outside in!
• Design from demand!

As you can see, a bit different from the original list. Still a bit long for my taste. The last two could, and perhaps should, be subsumed into the first. On the other hand, they are pretty important in their own right.

The re-imagined principles have important implications for methodology development. Here is a small collection of things that might become different if Agile was re-imagined using the principles above:

• More focus on building the right thing. Most Agile methods today, with the notable exception of Crystal, are primarily focused on building things the right way. User interaction design would become more important with this the re-imagined version of the principles. Systems Thinking would play an important role in designing solutions for customers.
• Studying and training would become more important. In some organizations it would become a regular part of the working life.
• Systems Thinking would be applied to software methodology design to a higher degree than it is right now.
• The idea that there are practices who are "best practices" regardless of context would be abandoned.
• Studying, measuring, and experimenting with work methods would become more common.
• Team structure and composition would change. We would probably move towards "solution development teams" rather than "software development teams".
• It is likely that software frameworks would become less popular, and reusable software modules and libraries would gain in popularity. If this trend is strong, it might affect which programming languages that are used.
• Widespread education in Systems Thinking might lead to an increased interest in functional programming languages.
• Many network organizations would have Intranet-based social networks. This would increase lateral interactions in the organizations.
• Some organizations would have software library repositories similar to CPAN and RubyForge. Though probably for other languages...
• Some organizations might go for Wikipedia style knowledge repositories. These would contain information about everything of interest to the business organization.

Or, something entirely different might happen...

Whether you agree with the ideas I have presented here or not, by reading this far, you have certainly exhibited a great deal of curiosity and tenacity.

Thank you!

Re-imagining Agile part 1: Why it shouldn't be done, and how to do it

There is a long running thread at LinkedIn, well over 200 posts now, about the principles of Agile software development. The issue is whether the principles can be improved, whether they should be improved, and how that might be done. I have written about the thread before, but there are still a couple of things mulling about in my head.

I got into a real writing fit this time, so I have split the article into parts. This is the first. More will follow.

I wrote in the thread that, technically speaking, I do believe the Agile principles can be improved upon. I also wrote that I believe it is a bad idea to try:

• I do not know anyone who has the mandate to change the basic principles of Agile. The original signatories might, but I doubt they want to.
• There is an existing body of work based on the current version of the principles. That body of work will not be redesigned just because someone releases The Principles of Agile 2.0. Scrum won't change, Extreme Programming won't change, the enterprise organizations that keep misunderstanding Agile won't change.

To put it bluntly: The declaration of principles are not, at this time, an effective leverage point. Changing them will not accomplish much. (At the time the Manifesto and the principles were written, they managed to accomplish a lot - more than I would have believed to be possible back in 2002 when I first began using Extreme Programming.)

On the other hand, re-imagining Agile is an interesting exercise. I like to tinker with systems, so let's do it for the fun of it, just to see where we end up.

If I were to design a framework for developing hyper-effective software development methods, where would I start?

I am a systems thinker, so I want to design from the outside in. What is the outermost layer of interest? The paradigm! Our paradigms, our worldviews, shape the systems we build. Paradigms are the tools we use to make sense of the world around us.

(If you are an Agile toolhead and like it that way, you can save yourself some trouble by not reading the rest of this. You will be confused, upset, and lose faith in what you are currently doing. Trust me, I went through it myself many years ago.)

Why should we begin with paradigms? There are three important reasons:

• All paradigms are models, and all models have limitations. If we understand the models we use, we can consciously select the thought model best suited to deal with a particular problem.
• If we understand our own paradigms we are in a position to detect when they fail us. This is a prerequisite for improving them, or switching to better ones.
• The currently dominant paradigm is seriously out of touch with reality. I'll write more about this.

First, let's look at some of the sources that influence the paradigm I want to use. Click on the picture to see a larger, readable, version.

Fig 1: Sources of the new paradigm

This is a simplified figure of course. It covers only literary sources, omits many important works, and does not show the relationships between them. Nevertheless, it does serve as a high level map of the ideas shaping the basic worldview I intend to use.

This is important: You can check the map! You can read up on the ideas and test their validity yourself! Thus, you can find flaws, or alternatives that suit you better, and improve on it. (Please tell me when you do - helps me improve my own paradigms.) There is nothing stopping you from developing many different maps, and pick the one most appropriate to the situation at hand. Technically, it is a paradigm transcendent meta-framework. I call it Strategic Navigation, because it was to a large intent inspired by Bill Dettmer's book Strategic Navigation.

Scientific Management, the currently dominant paradigm, asks you to take its ideas on faith. It claims to be the best, regardless of circumstances. And, despite the name, it isn't very scientific at all. Scientific Management used to be more scientific than it is today, but unfortunately, the initial curiosity that begat Scientific Management solidified into dogma very quickly. (Unfortunately, the same thing is happening with Agile today. This death of curiosity is often called "maturity".)

For brevity's sake, I will assume that you know the basics even if you haven't read the same sources. You need to know a little bit about Systems Thinking, TPS/Lean, Theory Of Constraints (TOC), Maneuver Conflict and Theory X/Y. I'll introduce you to some of it in this article, but do not expect a softly cushioned introduction. I'd need to write another book for that, or perhaps make a movie.

Given the paradigm in Figure 1, we can determine that the system we want to design Agile for consists of the following entities:

• Customers - This is usually a group of many different customers. Quite often, it is possible to divide this entity in several different sub-types with different properties.
• The business organization
• The subsystem in the business organization responsible for delivering value to a customer or type of customers.

I will deliberately refrain from describing the system in more detail than this just yet. We will look at a couple of different system designs later on.

Why do we need new ideas?

We don't need new ideas unless there is something wrong with the ones we already have. In the LinkedIn thread that inspired this article I suggested that it is better to redesign the enterprises to fit reality, than to redesign Agile to fit dysfunctional enterprises. The response was that if you have a Successful Large Enterprise, there is no need to change it.

That is of course true, but it does raise a few questions:

• Successful on whose terms? The owner's? (All of the owner's? Which owner groups?) The people who are members of the organization? (All of them? Only particular groups?) The customers of the organization? Depending on the situation, and the paradigms of the people involved, you may be able to satisfy all (win-win, perspective of bounty), or only a few, perhaps only one (win-lose, perspective of scarcity).
• What is success? Making as much money as possible? Hardly. Money is a tool. Having more money means "having a more powerful tool". If you have a lot of money but don't know what to do with it, you are pitiful, not successful.

Nor is growth a measure of success. Organizations grow in order to achieve their purpose better, and to enhance their chance of survival, but growth for its own sake is cancer, not success.

Death to the successful - How successful growth can turn into corporate cancer

Most business organizations do not survive long, even if they grow very large. The life-span for a large organization is about the same as for a human. While the average life-span of humans is growing longer, the average life-span of business organizations isn't.

It is difficult for an enterprise to be successful in the long term if it is pre-destined to have a short life-span.

There are plenty of life-cycle models that show the life-cycle of organizations. It usually goes something like this:

1. Start-up
2. Growth
3. Maturity
4. Decline
5. Death (or, with a few lucky ones, Reinvention)

These models often go into details of the characteristics of each stage, but they rarely provide insight into how this cycle can be prolonged, or even stopped entirely.

When you look a bit closer at the life-span of organizations, some things stand out:

• Some organizations last a lot longer than the norm. (Religious organizations, countries...)
• The short-lived organizations are functional hierarchies with Command & Control cultures
• The long-lived organizations have network structures, are purposeful organizations, or combine both features

Given the paradigm in Figure 1, this is not surprising. While an organization can croak for a variety of reasons, we can easily explain why hierarchical Command & Control organizations have a poor survival record.

This illustration shows how information flows in a hierarchical structure:

Fig 2: Hierarchical structures are prone to information overload

Because important decisions are taken at high levels in the organization, managers need to deal with more and more information the higher in the structure they are. This causes serious information overload, which gets progressively worse the larger the organization is.

Fig 3: Organizational complexity leads to paralysis

As Figure 3 shows, when the organization tries to defend itself against the information overload caused by its own growth, it actually makes the problem worse. Eventually, it grows so paralyzed it cannot adapt to changing external conditions, and kills itself off.

There are more problems related to information overload. For example, managers try to defend themselves against information overload by dealing with aggregated information. However, when you aggregate information, important signals will disappear in the random noise generated from other sources. For example, you can have a project portfolio that does very well even if one of the projects in it is in deep trouble. A manager that looks only at portfolio information will miss this.

A very serious problem pertains to how managers try to control their organizations in functional C&C hierarchies. Look at Figure 4. It shows various types of measures managers can take in order to change an organization.

Fig 4: Leverage points for intervening in an organization

Most managers don't even know this range of options exist. They focus on setting targets, the weakest intervention point. The result is bad management.

Fig 5: Managing by setting targets is bad management

You might wonder why setting targets is bad. Well, the assumption behind setting targets is that the people responsible for achieving the target are also the people who are in control of the results. This is not true. There are many factors, controlled by many different entities, that influence the results.

First of all, there are factors outside the organization itself. For example, you can deliver the same product or service to two different customers who have the same problem, and one customer will love you, the other will hate you, because the customers have different expectations. (Remember the brouhaha over the iPhone 4 antenna trouble just a short while ago. To most people it isn't a problem at all, to others it was the end of the world, or at least a sign of the impending doom of Apple.)

Of course, an organization that outsources a lot of work becomes more vulnerable to variation. This is rarely taken into account when outsourcing deals are made, but it is important to the performance of the organization outsourcing the work.

Second, there are internal factors. If the organization is a functional hierarchy there are many such factors (because of all the functional sub-divisions) and they are often in different control structures (different legs in the organization chart).

Fig 6: Most causes of results are outside the control of the responsible party

You get something like figure 6. You may be responsible for achieving a target, but you have very limited control over the factors influencing the results.

As a result, you get into a completely ridiculous situation. Mapping target levels to process behavior using a Process Behavior Chart, you get something like figure 7.

Fig 7: Targets drive dumb behavior no matter how you set them.

To make it worse, reward and punishment systems are often set up so that they are triggered by random variation in results, not by how well people have actually done the work. Random rewards and punishments are a great way to drive people a bit nutty, and make them do crazy things.

Add to this that scientific research shows extrinsic rewards, like money, has a tendency to short-circuit the brain. The scientific community has known about this since the sixties. Thanks to modern neuroscience we know a bit about why this happens: Extrinsic rewards trigger the pleasure center in the brain and deactivates the center responsible for socially well adjusted behavior. Pretty much the same effect cocaine has. (Read up on it: Drive by Daniel Pink, Punished by Rewards by Alfie Kohn.)

I won't write much about performance evaluations here, but you might want to check out this link. Suffice it to say, most performance evaluation systems in use today tend to do more harm than good to the companies that use them.

If you have read this far, you might start to think people are terminally stupid. Not true! However, people adapt to the systems they are part of. If the system is built based on the assumption that people are selfish and stupid, then you will get selfish and stupid behavior.

Theory X - The root of all evil

As I have mentioned, most business organizations are based on Scientific Management. Scientific Management was created by Frederick Taylor. Taylor believed workers are lazy, stupid, and motivated only by money. (Don't take my word for it. Read The Principles of Scientific Management.) He created a management paradigm designed to force lazy, stupid, disinterested people to work.

Theory X, formulated by Douglas McGregor, is a theory of human motivation that encapsulates the views of Taylor and many, many, more people. As it turns out, people who have Theory X mindsets will design organizations a certain way. See Figure 8.

Fig 8: Theory X affects how organizations are designed.

When you have an organization based on Theory X, the members of that organization will fear being punished for failure. However, only failures of commission (you do something and fail) are punished. Failures of omission (you didn't do something you ought to have done) are usually not punished. The result is that you get very risk averse organizations. People in them will consistently miss opportunities, and fail to correct systemic problems. (Edwards Deming estimated 94% of all problems in businesses to be systemic. I believe his estimate was too low.)

People will also be tightly controlled, which makes life excruciatingly painful. Eventually the pain goes away, but only because people get numb and institutionalized.

Theory X is self-fulfilling: Scare people and control their every action, and they will lose their motivation, they will behave stupidly, and the only thing motivating them to work will be the pay-check.

At this point you might wonder: What about the Successful Large Enterprise? How can a company grow to giant size and make tons of money if it is a wreck by design and run by zombies?

First of all, if you look at the situation right now, most companies are built on the same ideas and operated in the same manner. Therefore they perform about equally well.

Second, if you look at how business organizations have developed over the past 150 years or so, they could have developed very differently. For example, Henry Ford had two different plants operated in slightly different ways. He used one of the plants as a template for how to develop manufacturing at Ford. The plants were visited by a group of Japanese engineers and business people in the late forties. They were more interested in how the other plant worked, went back home, and built Toyota. (The full story is in Lean Cost Management: Accounting for Lean by Establishing Flow by James Huntzinger.)

The way Ford chose to operate was a success for customers and shareholders, but not for the people working in his factories:

Despite higher wages Ford's new system suffered from stupendous labour turnover. Newly hired workers lasted an average of only three months. Many walked off the job without any formal notification, and were presumed to have quit after missing five days of work: the notion of the 'five-day man' was born and accounted for 70 percent of the workers leaving Ford. Mass production systems were and are monotonous, demoralizing places to work. Trade unions grew out of the 20th century systems of mass production; current-management union practices serve to maintain the dysfunctional relationship. The relationship won't change until the system - the way work is designed and managed - changes too.

Systems Thinking in the Public Sector, by John Seddon

John seddon is right. Here is a personal experience:

After the financial crisis in 2008 about 10,000 people lost their jobs in the region of Sweden were I live. Many of those who lost their jobs did so quite unnecessarily. To put it bluntly, it is easier to fire people than it is to figure out how to be profitable in a recession. Still, for many companies it is quite doable to move forward in a recession because the competition slows down and miss even more opportunities than usual. I contacted a representative of the union I am a member of, and asked if they were interested in trying to prevent some of this. I proposed that the union should collaborate with management in order to figure out how to minimize the damage from the recession, or even take advantage of it.

I shouldn't have bothered. The response was what you would expect:

"We don't do that! We are not interested in doing anything until after our members are laid off."

This is exactly the dysfunctional behavior John Seddon writes about.

Sometimes I wonder what would have happened if Henry Ford had been more interested in that other plant.

Why management often looks like a train wreck

In 1841 there was a train crash in the United States. To prevent similar crashes from occurring a system for controlling operations by dividing responsibilities and authority, and by having a system of reporting and checks. This was the origin of the hierarchical organization chart. (Systems Thinking in the Public Sector, p. 48, by John Seddon)

What if that train crash had not happened? What if people had realized that a system designed to prevent low probability high impact events like train wrecks, are not very good for running normal high probability, low impact business operations? What if people had realized that a system designed to prevent one kind of problem often causes other kinds of problems?

The secret of enterprise success: Path dependency

Small events can cause very large effects in a certain type of systems: Systems dominated by reinforcing feedback loops. Expanding markets are dominated by reinforcing feedback loops, and it was in this type of environment the large enterprises we have today were born.

It is not always the best system that wins in this type of environment: The QWERTY keyboard is designed to slow typists down (because early typewriter mechanisms could not keep up with the speed of typists), VHS beat Betamax, and a crackpot system for business organization and management beat better systems.

The phenomenon is known as path dependency. It explains how past events affect the future. For example, Java was designed to appeal to C++ programmers. C++ was designed to appeal to C programmers. Thus, the design of C and C++ have influenced Java. Java has had a lot of impact on the design of C#. That does not mean Java and C# are the best language designs possible. People use it because other people use it.

It is the same thing with management paradigms. Theory X reinforces itself, so once it became dominant, the ball just kept rolling. It does not mean Theory X is a correct description of human nature. Nor does it make Theory X suitable as a foundation for building business organization.

All it means is that shit happens. (Path dependency is thoroughly explained in Business Dynamics by John Sterman.)

Good things happen too: Theory Y and Agile

Fortunately, good things happen too. So far, I haven't really covered anything new. I haven't even covered all the bases. For example, Cost Accounting was developed based on ideas stemming from Scientific Management. Cost Accounting is one of the main drivers of poor decision making in organizations. I won't go into the reasons, but you can check it out yourself. The references are in Figure 1.

The ideas I have written about here, Systems Thinking, variation, etc., are ideas that prompted Agile. (If you don't recognize them, I suggest you read a few books by Kent Beck, Jim Highsmith, Allistair Cockburn and other people who created Agile. Pay attention to the reference lists at the end of the books.)

There is another idea that is fundamental to the development of Agile. It is called Theory Y. Theory Y was developed by Douglas McGregor, who also formulated Theory X. Like Theory X, Theory Y is self-fulfilling (it becomes true if you act as if it is true), and it is self-reinforcing. However, the basic assumptions about human nature are the opposite of Theory X. If you believe in Theory Y, you will develop organizational structures and management methods that are completely different from the structures in Theory X companies. Figure 9 shows Theory Y

Fig 9: Theory Y - The foundation of Agile and the next generation of business organizations

Note how the job of managers change. Their job is now to create the right conditions for people to enjoy and excel at their work. To do this, they need to understand how to use, and not to use, all the levers shown in Figure 4.

It is a completely different way to manage, and you know what: A lot of people are to set in their ways to make the change! If you are a lord in a feudal society, a shift to democracy would rob you of your position and power. In a similar way, if you are a manager in a Theory X organization, you are quite likely to lose position and power if the organization transforms to Theory Y.

In some Theory Y organizations, like Ricardo Semler's Semco and W. L. Gore & Associates, members (they don't have employees in the traditional sense), elect their own bosses. (Terri Kelly, the CEO of W. L. Gore & Associates was elected to her position.) How many bosses do you know that would be elected to their position, if the people voting were the people working for them?

There are Theory Y organizations that are not quite so democratized, for example Richard Branson's Virgin Group. Still, it is a bit of a stretch imagining a C-level executive from GM doing very well at Virgin. GM managers and Virgin managers make people laugh, but for entirely different reasons.

Agile is designed to work in Theory Y organizations. It is designed to enable Theory Y organizations to vastly outperform Theory X organizations!

Theory X and Theory Y are fundamentally incompatible. Going back to the LinkedIn thread for a moment: The change that must be made make Agile more easily adopted by large enterprise organizations is to replace Theory Y with Theory X. Of course, if you do that, Agile won't be Agile anymore. We'll be back to old waterfall methods, or worse.

Agile's big mistake

When I became interested in Agile in 2002, I became enamored by the possibilities. Like early evangelists, I believed Agile would be like an asphalt flower: The roots would spread, cracking the asphalt, opening ways to transform companies in all areas: organizational structure, management practices, strategy, economic models, sales and marketing. Even back then, I dimly grasped that for a major change in software development methodology to work, other parts of the organization must change too.

Like the early evangelists I believed the proof would be in the pudding: Agile produced results, great results, so of course management would be delighted.

In retrospect, a pretty naive idea. Agile was, and is, a misfit with most companies. For awhile, that was difficult to accept. Given the current system, managers trying to adopt Agile methods take great risks. Agile methods tend to expose the problems proliferating in other parts of the company, so a manager supporting Agile can easily make a lot of enemies. (This is the same problem managers have when they try to adopt Lean, TOC, and other systems that can improve the performance of their companies.)

However, I have changed my own perspective a bit: Agile won't be successful by being adopted by dysfunctional dinosaurs. Agile will be successful by being adopted by the new breed of companies that will replace them.

Agile's big mistake was designing from the inside out: trying to improve a part of a system that just isn't designed for that kind of improvement.

So, if we want to improve Agile, we need to design from the outside in. We need to improve the fit with the next generation of business organizations. What will that next generation look like?

Watch out for Part 2!