Re-Imagining Agile part 2: Designing from the outside in

A business that makes nothing but money is a poor business.

– Henry Ford

In the first part of the Re-Imagining Agile series I wrote about some of the ideas Agile is based on. The most basic one is a theory about human behavior and motivation, Theory Y.

I wrote that most business organizations are designed based on a different set of assumptions, Theory X. Moving to Agile therefore represents a paradigm shift, a change in how we think.

Fig 10: The paradigm shift.

Figure 10 illustrates the shift in paradigms. As you can see, the shift is much more extensive than switching from Theory X to Theory Y. You might wish to compare this figure with the reading map in Figure 1 (in Part 1 of this series).

While it is difficult to make the transition, it is quite possible. One organization that has made this shift is the U.S. Marine Corps. They did something really sneaky: They kept the name Command & Control, but they changed the definition of the word control!

...decentralized control works from the bottom up. Command is the exercise of authority and guidance, and control is felt as feedback about the effects of the action taken because thinking is required at all levels.

Tactics, MDCP 1-3 - The U.S. Marine Corps manual on Tactics

It is my belief that what has enabled the Marines to make the paradigm shift was that they had pretty strong Theory Y leanings even before.

For now, I am going to leave the question of how to make an organization, even a large organization, make the basic paradigm shift. Instead, I will focus on what we want to achieve. As Dee Hock, who created VISA, put it, we must imagine what ought to be!

Decisions - the Power of the OODA Loop

We began by defining the paradigm, the system of thought, we need to re-imagine Agile. What is the next layer? We know that we want to build a system based on Theory Y. We know we want to design from the perspective of the customer, but it is not quite that easy. A business organization must make some pretty complicated decisions, like who the customers are, and what particular needs the company should satisfy, the nature of their relationship with customers, how to organize to utilize their resources as well as possible...

Decisions, decisions! Everyone in a business organization must make a lot of decisions, every day. What if we could have a common framework for all decisions throughout the organization? What if the CEO, a software developer, an accountant, and sales person, everyone in the company, including the people who empty the waste baskets, all had a common framework for making decisions?

The purpose of such a framework would be to improve the quality of decisions throughout the organization. Thus, the framework must make it easy to make good decisions, and make them quickly.

As it turns out, there is such a framework. It was developed by U.S Air Force Colonel John Boyd. Originally it was used to describe the decision processes of fighter pilots, but it soon gained traction as a generic model for decision making. The model is called the OODA Loop, and it is at the core of John Boyd's strategy meta-framework, Maneuver Conflict.

Fig. 11: The OODA Loop - An effective decision model

Figure 11 reveals the OODA Loop in all its gory glory. When I first saw it I thought, "What the #%&@ is that?" so don't be taken aback if it looks a bit complex at first.

I will not go into details about how to use the OODA Loop, but I will mention that an important facet is the ability to change speed and direction very quickly. In other words, to be Agile!

Few companies have strategy cycles shorter than a year. Using the Strategic Navigation framework, a business implementation of Maneuver Conflict, strategic cycles for large companies can be as short as a week.

Just like Agile methodologies allow development teams to use short iterations and short delivery cycles, so can business strategists compress strategy cycles with Strategic Navigation.

Just to illustrate the advantages fast strategy cycles can bring:

One company that is very good at compressing their strategy cycles is Apple. First they launched the iPod, then they used the popularity of the iPod to launch iTunes. iTunes, with the addition of the AppStore, became major incentive for customers to buy the iPhone. iTunes and the AppStore, with the addition of the new Bookstore greatly enhance the attractiveness of the iPad. Concurrently, they use Mac computers to increase the value of iPod/iPhone/iPad/iTunes, and iPod/iPhone/iPad/iTunes to increase the value of Mac computers. For example, the video capabilities of iPhone 4 fits neatly with the iMovie program included with all Macs. Most of the competition is far behind, trying to copy the iPhone/iTunes combination, while Apple continuously shapes and reshapes the market.

Since we have a spiffy new paradigm at our disposal, let's use it. We switch from Maneuver Conflict, where we found the OODA Loop, to Queueing Theory, where we find Little's Law:

t = I/T

Where:

t is lead time
I is the number of items in a process
T is the rate at which items are completed

As you probably know, Little's Law is fundamental to Agile software development. It is a natural law, and if Little's Law didn't work, Agile wouldn't work either. (Suggested reading: Lean Software Development and Implementing Lean Software Development by Tom and Mary Poppendieck; Product Development Flow by Donald Reinertsen.)

When I became interested in Maneuver Conflict, and saw the OODA Loop , something went BANG!!! in my head, and the lights turned on.

I suddenly realized that Little's Law is vitally important to the OODA Loop too. The arrows in the OODA Loop diagram show how information flows when we make decisions. The Post-It notes on a kanban board also show how information flows. They are both governed by the same basic law, Little's Law!

We need to make good decisions fast. Little's Law gives us an idea about how to accomplish that: We can either reduce I, or increase T. How do we do that?

We can reduce I in top levels of an organization by making as many decisions as possible at lower levels of the organization. At the same time we engage more people in thinking, and thus increase T.

Quick switch to our Systems Thinking hat: When we decentralize decision-making we shorten information feedback loops, which shortens delay in the loops, which increases the effect and reliability of the feedback. If you look at Figure 4 in Part 1, you can see that we can improve two high impact leverage points, the information flow structure, and the system structure, in one fell swoop.

Looks like Maneuver Conflict, Queueing Theory, and Systems Thinking all lead to the same conclusion. Great!

Here is one implication: If the CEO of your company understands the OODA Loop, and you, a software developer, understand the OODA Loop, you know have a common frame of reference.

Because everyone in the organization can use the OODA Loop as a frame of reference for how they make decisions, everyone in the organization has a shared frame of reference for understanding each other.

Once again, when I write "the OODA Loop", I do not mean just the diagram. I mean the whole package that goes with it.

We now have a starting point for developing an organization designed to take full advantage not just of of Agile software development, but also of modern ideas about how to organize, modern economic ideas, modern ideas about product development...

Figure 9 (in Part 1) showed some of the implications of Theory Y:

• Workers must be encouraged to communicate vertically and laterally. (No organizational silos.)
• There is no need for elaborate hierarchies of control and authority
• Workers should have a say in decisions that affect them.
• Failures that result in learning should be rewarded. (No fear culture! For brevity's sake, I will not go into this point any further.)

We can use the OODA Loop, Little's Law, and the list above to begin designing an agile organization.

We need to design the organization so that it consists of small, autonomous units. That way, the organization can grow without overloading the OODA Loops of top management. At the same time, the organization can quickly reconfigure itself when it needs to do so.

We need to weigh economy of scale against economy of speed. When we are the slightest bit uncertain, we'll choose speed. Speed is very important when the environment changes quickly. And we want to do more than to be able to respond quickly, we want to actually shape the environment. That requires the ability to go fast, and to change speed and direction very quickly for the whole organization!

Judging from our sample companies in Figure 1, we want to keep the main organizational units no larger than 100-200 people. Beyond that, we will loose to much agility. Of course, this is a very general guideline. For example, if your company is an industrial company, you may need to have some very large main units, like a large factory. Still, the principle holds, and even within a large factory, we can strive to make value streams interfere with each other as little as possible. (This is what Lean does with takt time. When variability in demand is high, as in software development, it can be better to separate the value streams as much as possible.)

What would this look like? Well, there is no absolutely fixed structure. You can't have a fixed organization chart, because the structure keeps changing. (I am not joking. For example, Semco, one of the model organizations in Figure 1, does not have organization charts for precisely this reason.) But you can show the basic principle.

Fig 12: Simplified network structure

Figure 12 shows the principle. Keep two things in mind: The figure shows a snapshot in time, and it is highly simplified.

We are designing from the outside in, so let's begin at the point where the business organization has contact with the customer. I am going to assume that the organization builds custom software of some kind. The particulars are different for other kinds of organizations, but the principles of designing from the outside in and designing against demand, i.e. from the need of the customer, remain the same.

I am also going to push the principles of Theory Y, pull and self-organization to the max here. Remember, we are designing what ought to be, not trying to make minor improvements to what already is. Agile is not faster waterfall, so why limit ourselves when we are designing an organization ideal for taking advantage of Agile?

Teams are built around value streams

The modern industrial organization is a vast complex of interdependent relationships, up, down, across, and even "diagonally." In fact, the system is so complex that only collaborative team efforts can make the system work effectively. It is probable that one day we shall begin to draw organization charts as a series of linked groups rather than as a hierarchical structure of individual "reporting" relationships.

The Human Side of Enterprise by Douglas McGregor

Let's assume initial contact is between the customer and a sales-person. The sales-person would be trained in a sales method compatible with Agile software development, for example the Prime Process. The Prime Process has roots in Systems Thinking, and just as important, it is an explicitly ethical sales method with focus on delivering business value to the customer. (See Mastering the Complex Sale by Jeff Thull.)

If you are an Agilist, you and a Prime Process sales person do have a lot in common: shared goals, and shared assumptions about how to achieve them.

The sales person posts the status of negotiations with the customer. Skills needed for a sales team will also be publicly posted. This will most likely include people with design skill and technical skills. People interested in the job can sign up to work on the sales team.

Because people sign up, instead of having someone assign them, we reduce management work. We also eliminate the need for costly systems to track which people are available. Sales people must know whether people are available to do work, so the company will probably have a kanban-like system to ensure that sales people don't over-commit.

Signing up might be as easy as moving a kanban card representing you, the developer, from a resource pool area on a whiteboard, to a project team area on a different board. Put the boards in the cafeteria, and everyone will be able to track what everyone else is doing at all times.

If the sales team does not make the sale, everyone just moves their cards back to the pool. If a sale is made, the team continues to work with the customer. The sales person may remove himself from the team in order to continue selling, but some of the people will remain, which provides continuity for the customer, and for the team itself. If more diverse skills, or more manpower, is needed, a message about the open positions are posted on the board.

You might ask what happens if people don't want to sign up. That, I am afraid, is a Theory X question. Ask instead: Given that people want to work, and want to do good work, what could prevent them from signing up? If you come up with a real obstacle, fix it. If something is preventing you from fixing it, fix that first!

What if there isn't enough people, or anyone with the right skills, available within the company? Post the job openings to a computerized version of the board that is available to the entire business organization. The team can also contact other companies in the same organization directly, or through Power Groups. (A bit more about those later.)

Note that the team does not necessarily have to bring people from other parts of the organization in. If it is advantageous to do so, the team can temporarily move itself to another company for the duration of the project. Profit sharing schemes for handling this are pre-arranged, so no complicated negotiations between business units are necessary.

By this time you probably think I am off my rocker, but there are real organizations that exhibit precisely this kind of flexibility. A couple of examples:

A U.S. Marine Corps team can decide to move from one chain of command to another if it makes it easier for the team to accomplish its mission.

I am a member of Business Network International (BNI), a business referral network. BNI has about 125,000 members. I belong to a team named Gothia Towers. Today, I had lunch with two BNI members from a different team. They invited me to visit their BNI team next Friday, when they have their weekly business meeting. I accepted. In effect, we took a lateral contact, and even temporarily rearranged the team structures.

If you had suggested to the team members who invited me that they should check with the team management first, they would look at you as if you were a bit funny in the head. They would be right. Their team leaders would be justifiably annoyed if they did. (So I am very glad the thought never crossed your mind.)

If the right people cannot be found within the organization, the team can decide to hire people temporarily. They can do this because:

• They have an economic model to guide them (See Product Development Flow by Donald Reinertsen)
• They are not constrained by a pre-determined budget (See Beyond Budgeting by Jeremy hope and Robin Fraser)
• They are self-organizing

A team pulls resources if and when it needs them. If the team grows too large, it restructures itself into sub-teams.

Fig 13: Linked team structure and communications pathways

As figure 13 shows, each person with leadership responsibilities is a member of two teams. This is different from managers in a hierarchical organization having meetings together. The leaders in Figure 13 work together, ensure that the teams they lead collaborate effectively, and make trade-offs where necessary.

As I have already mentioned, anyone in a team is free to contact any other team without going through a vertical hierarchy. There is a strong emphasis on lateral communication.

A third thing that is different, and once again borrowed from the U.S. Marine Corps, is the idea of the Directed Telescope, or pull communications.

In a hierarchical structure, information is usually pushed upwards, through standardized reporting procedures. A network structure will have similar procedures for pushing information inwards. (Though the type of information, and the amount of information, is very different because the paradigm governing what is considered important information is different.)

There is another way to get information: Pulling it. That means someone at the center actively goes out and looks at the situation in order to understand it. This is sometimes known as Management-by-Walking-Around. Lean folks call it genchi genbutsu, “actual place, actual thing”, to go and observe in order to fully grasp a situation. ("Grokking", if you have read Robert Heinlein's Stranger In A Strange Land.)

One reason why pull communications are under-used in hierarchical organizations is that managers are already swamped by information pushed at them. In the network structure we are designing, we do not push nearly as much information to managers. We also present information in different ways, for example by using process behavior charts or TLTP diagrams. This makes it easier for centrally located managers to pull information when needed.

Note that the structure in Figure 13 allows both independent action and close collaboration. The same linked team structure is used throughout the entire organization.

Depending on what they do, some teams last only a short time, other teams may be very long-lived. For example, a team responsible for the grocery section in a supermarket would never disband, even though membership would change over time. (Remember, we are talking about a general structure for business organizations, not just a structure for software development teams.)

In some business organizations, like Semco and W.L. Gore & Associates, team leaders are elected by their teams. It might be a good idea to require elected leaders to take advanced training in the leadership skills mandated by the basic paradigm. Of course, everyone in the organization should have basic training, and plenty of opportunity to learn on their own if they so choose. It might look like Figure 14.

Fig 14: Mandatory training of elected leaders

Companies

For the purpose of the model, I have assumed that the organization is fairly large, and that the basic business unit is an independent company. A structure like this provides economic protection. If one company fails, other companies will be affected as little as possible.

It is also possible to have other companies absorb people from the failed companies, or use the freed manpower to start new businesses. This is a strategy that the Virgin Group has used successfully for many years. (See Business Stripped Bare, by Sir Richard Branson.)

A company is responsible for one or more value streams. Figure 15 shows the principle.

Fig 15: A company is a linked team structure with the responsibility for managing one or more value streams

I mentioned a size limit on companies. Keeping companies small keeps information loads under control. The exact size limit depends on what business the company is in.

When a company grows to its limit, part of it is split off to form a new company. The Virgin Group employs a strategy where the new company must find a market for itself that is different from the markets of all other companies in the network. That is why Virgin is so diversified: Music, hotels, spaceflight, soft drinks, etc. This prevents different companies in the network from competing with each other.

Traditional enterprises also divide themselves into companies, but they do it in a different manner. It is quite common for an enterprise to start a new company for the purpose of executing one step in a process. The result is a chain of tightly interdependent companies. If one company fails, so does the process, and the other companies in the chain can't get the revenue they need to survive. This makes all companies in the chain very vulnerable. To make it worse, the companies usually use Cost Accounting practices, which means they all strive to be as profitable as possible, which means they have to fight each other.

It is very important to avoid this trap, and set companies up so that they really are built around end-to-end value streams.

Power Groups

As I showed in the section about teams, it is quite likely that these companies have opportunities to collaborate now and then.

Sometimes team level collaboration is not enough. For example, companies might share information on a continuous basis, or work together to execute major operations.

When a company believes it can benefit from formal collaboration with one or more other companies, it can attach itself to a Power Group. Power Groups share information, and may help each other in other ways, as they see fit. Note that a company decides for itself which power group, or groups, it wishes to apply to. The power group then decides whether to accept the application. A power group may of course elect to invite a company.

The Core

There is no "up" in a network organization of this type, and thus no "top management". However, at the center, there is a core, and a core team. The core team is responsible for the system as a whole. Following the same structural pattern as before, most members of the core team will also be leaders of companies, service units, or power groups.

The core team ensures that the organization moves towards the overall system goal. Depending on the organization, they may be the people setting the systemic goal, or not. For example, at the Virgin Group, this is the people ensuring that the brand values are adhered to throughout the organization. Using Maneuver Conflict terminology, they are responsible for the organization's Noble Vision.

The Noble Vision is an idea that unites not only the people within the organization, but also the organization and its customers. Think about Apple brand loyalty, a football team and its fans, and you'll get the basic idea. There is also a moral dimension to the Noble Vision concept, but I will not go into detail here. Suffice to say that the Noble Vision must be something that you and your team mates believe is worth striving for.

One very important service that should be handled by the core in most organizations is basic management training. Every leader in the organization, whether a Scrum Master or the CEO, must have the same basic training. Look at the Orientation step in Figure 11. There must be a shared base here, or it will be impossible to develop the trust and understanding necessary for decentralized command.

A related service that might benefit from centralization is management consulting. For example, the Virgin Group has an internal management consultant organization for the purpose of helping to solve thorny management problems.

A basic rule is to centralize services that benefit from economy of scale, and decentralize services that benefit from economy of speed.

IOHAI

I have mentioned the need for leadership training a couple of times in this article series. When John Boyd developed Maneuver Conflict and the idea of a Noble Vision, he also conceived a leadership model, IOHAI. The IOHAI model is a framework for training leaders that can help move an organization towards a Noble Vision.

Boyd considered IOHAI to be the key to enabling vitality and growth. It is certainly not the only management or leadership framework that can be used, but it is a very useful one. It incorporates ideas of paradigm transcendence and continuous learning.

Fig 16: IOHAI - Theme for vitality and growth

I wrote my book Tempo! so that it can be used as an IOHAI training manual for leaders at all levels in a business organization. Once again, I will not go into details here, but when you dig down into IOHAI, it becomes very practical and action oriented.

Design notes and useful Intermediate Objective Maps

We can summarize our design goal for the organization, and the necessary conditions that must be satisfied in order to achieve the goal. Figure 17 shows what that might look like.

Fig 17: A generic organizational IO Map

Though I did not write about it, I used Self-Determination Theory (SDT) as a guide when developing the organizational structure. Figure 18 shows how, according to SDT, intrinsic motivation can be stimulated.

Fig 18: Self-Determination Theory - The building blocks of intrinsic motivation

As it turns out, most processes have a lot in common. We don't need a template specifically for how to construct software development methodologies. We can construct a generic template for process design, and use that to design every process in the organization. Figure 19 shows what such a template might look like.

Fig 19: An IO Map for process development

Using the same basic ideas for designing all processes is very important. If you have a mismatch in design philosophy, the processes will not match each other very well, which will lead to a lot of waste of effort and time, and reduce the quality of the end result.

Agile principles re-imagined

This article series was inspired about a discussion in a LinkedIn forum about Agile principles. Given what has been done so far, what about the current Agile principles? Will they stand as is, or can they be improved upon? Yes, I believe they can. (As you may recall from part 1, I answered the question "should they be changed?" with "No!". But I answered "can they be changed?" with "Yes!")

Let's begin by defining the word "principle". It was clear from the LinkedIn forum that the word means different things to different people. I will go with the definition in my dictionary:

principle - a fundamental truth or proposition that serves as the foundation for a system of belief or behavior or for a chain of reasoning

There are 12 Agile principles, which I think is a bit much. The principles themselves are not simple. In addition to being too many, several are unnecessarily wordy.

Ironically, principle 10 is:

Simplicity--the art of maximizing the amount of work not done--is essential.

Some of the principles are best described as opinion, plausible on the surface, but unsubstantiated:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

The above sparked a lot of conversation in the LinkedIn thread. It is clear that it leaves many considerations out of the picture, and that face-to-face conversation is not always the best alternative. For example, if the information conveyed is very complex, it may be necessary to write it down in addition to talking about it. In many situations, face-to-face conversation is not possible.

Finally, some things are just wrong:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Why not reflect and fix immediately when a problem crops up? I am for regular introspections, but I believe it should complement immediate stop-and-fix, not replace it. Lean companies have andon and kaizen, not one or the other.

Here is my attempt at Re-imagining the Agile principles:

• Study and apply the paradigm!
• Be curious!
• Be tenacious!
• Design from the outside in!
• Design from demand!

As you can see, a bit different from the original list. Still a bit long for my taste. The last two could, and perhaps should, be subsumed into the first. On the other hand, they are pretty important in their own right.

The re-imagined principles have important implications for methodology development. Here is a small collection of things that might become different if Agile was re-imagined using the principles above:

• More focus on building the right thing. Most Agile methods today, with the notable exception of Crystal, are primarily focused on building things the right way. User interaction design would become more important with this the re-imagined version of the principles. Systems Thinking would play an important role in designing solutions for customers.
• Studying and training would become more important. In some organizations it would become a regular part of the working life.
• Systems Thinking would be applied to software methodology design to a higher degree than it is right now.
• The idea that there are practices who are "best practices" regardless of context would be abandoned.
• Studying, measuring, and experimenting with work methods would become more common.
• Team structure and composition would change. We would probably move towards "solution development teams" rather than "software development teams".
• It is likely that software frameworks would become less popular, and reusable software modules and libraries would gain in popularity. If this trend is strong, it might affect which programming languages that are used.
• Widespread education in Systems Thinking might lead to an increased interest in functional programming languages.
• Many network organizations would have Intranet-based social networks. This would increase lateral interactions in the organizations.
• Some organizations would have software library repositories similar to CPAN and RubyForge. Though probably for other languages...
• Some organizations might go for Wikipedia style knowledge repositories. These would contain information about everything of interest to the business organization.

Or, something entirely different might happen...

Whether you agree with the ideas I have presented here or not, by reading this far, you have certainly exhibited a great deal of curiosity and tenacity.

Thank you!