Going agile the wrong way - How enterprise adaptation hurts agile software development

The topic of how to make agile and enterprise organizations fit together better has garnered some interest. No matter what your position on the issue, if you blog about it, and tell me, I will list a link to your post here. I do believe this is an important issue:

• Pascal Pinck wrote some thoughtful comments on his blog Hanzatsu.

Now on to my post:

There is a rather active thread in the Agile Alliance forum on LinkedIn regarding the Agile Manifesto and whether it is in need of an overhaul. The premise is that enterprise organizations have been slow to adopt agile methodologies, and therefore basic agile principles should be revised in order to make adoption easier.

I disagree strongly. While I can see ways in which the principles behind the Agile Manifesto can be improved, I believe increased enterprise compatibility is the wrong goal to set.

Here is one of my posts in the thread:

@Alan - I liked the blog post. (Alan Shalloway's blog post is here.)

@Everyone - I looked through the reference sections of two of my oldest books about agile this morning: Beck's eXtreme Programming and Highsmith's Agile Software Development Ecologies.

Both Beck and Highsmith were inspired by Systems Thinking. (The 2nd ed. of eXtreme Programming explicitly mentions TOC as a source of inspiration.)

That was a good start, but somehow it never caught on. Instead, the focus turned inwards, to the team. Interaction between the team and its super-system has received very little attention.

Highsmith made the observation that while agile can fit a broad range of organizational cultures, there is one type that does not fit very well at all: Command & Control culture. Highsmith was correct. Agile and C&C culture fit very poorly. This is partly because of different mind-sets, but also for structural reasons.

C&C structures are hierarchical. Hierarchical structures cannot accommodate self-organization and mission orientation very well. Agile teams are supposed to self-organize. They are also supposed to be mission oriented. (C&C hierarchies and mission oriented organizations have incompatible command structures - commands flow in different directions during missions/projects.)

Thus, Agile teams make a good fit with Three Ring organizations (Semco), Lattice organizations (Gore & Associates) and Boydian networks (BNI is the closest I can think of in the business world at the moment), but a lousy fit with traditional C&C hierarchical organizations.

There are also, as Alan pointed out in his blog post, issues with flow. Agile teams are (supposed to be) geared to global optimization of flow. C&C hierarchies are intended to achieve local optimization of resource utilization. (Interestingly, this means Agile projects must execute within the context of portfolio management, something many enterprises do very badly. There are some brilliant exceptions, of course.)

C&C hierarchies have worked well for many years, but ultimately they are a dead end. When the complexity of the environment and the rate of change in the environment increases, C&C hierarchies find it more and more difficult to cope. No big surprise, because C&C hierarchies are designed for relatively static environments. A bit simplified, they are built to sacrifice economy of speed for economy of scale.

Today economy of speed is gaining in importance. Economy of scale is losing importance. (Toyota vs. most other car companies, print-on-demand vs. traditional publishing, fashion industry...)

Trying to adapt agile to fit C&C hierarchies will inevitably rob agile of the properties that make agile competitive in the super-system outside the enterprise. Most (but not all) enterprises are a poor fit with agile and with their own super-system.

Agile is supposed to be adaptive to customer demand. That does not mean agile should be adaptive to enterprises that are themselves not very adaptive to customer demand.

Trying to do both will lead to a failure to achieve either.

BTW, there are two things I would have liked to see in the Agile Manifesto: Explicit commitments to continuous learning and Systems Thinking.

Here is a follow-up to the first post:

It is worth noting that the Agile Manifesto is not a declaration of agile principles per sé. The principles are here: http://agilemanifesto.org/principles.html (Until this point, no-one in the thread had made the distinction between the text in the manifesto and the principles the text is based upon.)

If you look at the principles you can easily see there is a problem with fitting agile to C&C, for example:

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

C&C ideas about motivation are Tayloristic. This is reflected in companies in various ways, for example by various kind of monetary reward programs. Today many of these ideas have been disproved by scientific research. For example we know today that when you offer monetary rewards to knowledge workers (CEOs as well as software developers) you stimulate a brain center, Nucleus Accumbens, the pleasure center. And we know that when N. Accumbens gets stimulated, the ability to solve complicated problems drops. A lot. (Some books that touches on the subject: Sway, Predictably Irrational, Brain Rules, Punished by Rewards. I haven't read Daniel Pink's latest yet, but it looks promising.)

About "trust" in the second sentence: In addition to achieving economy of scale an important reason for using C&C hierarchies is to impose narrow spans of control for individuals, and enable extensive control mechanisms. You do this because you do not trust subordinates.

The problem is that this environment is not only engendered to replace trust with control, it also actively reduces trust in various ways. The division of labor model, as well as authority flow and information flow structures make it difficult to build trust among people in C&C hierarchies.

For example, compare a department in an enterprise with a BNI team. (BNI is a large business referral network, about 125,000 people. I am a member.):

In a BNI team leadership rotates. All members take turns leading the team. We have a shared common goal, designed so that each team member benefits from helping other members. We are evaluated on how much we support other members. Team members do active trust building exercises, and actively seek opportunities to get to know each other outside of team activities.

We do these things because we need to trust each other to be effective. Compare this to, say, the forced ranking systems used in many enterprises. See http://kallokain.blogspot.com/2009/06/performance-evaluations-business.html

Which type of environment do you believe to be more conducive to effective knowledge work and close collaboration? Which environment do you believe is most effective in motivating members of the organization?

The agile principles can be improved upon. The problem is that if you improve them to be more in line with science (physics, queueing theory, statistics, neuroscience, psychology, systems thinking), you make agile _less_ compatible with most enterprises.

If you improve the principles for the purpose of increasing compatibility with enterprises, you will have to remove a significant portion of the scientific basis. You will also make agile less responsive to customer's needs and desires.

The solution, I believe, is to redesign the enterprises. Most enterprises are built on old paradigms. Those paradigms have become obsolete. They are no longer compatible with the environments in which the enterprises exist.

First Steps: Starting up a book writing project

I have recently begun working on my new management book project. Translating Tempo! into English is my top priority, but the new book will be based on  interviews to a large extent, and I do not want to pass up good opportunities when I get them. I expect to do several interviews in parallel with working on the Tempo! translation.

I won't go into the topic of the new book, but you might find it interesting how I go about working on it.

When I began thinking about a new book I started with the question:

• "Who am I writing for?"

Once I had an answer to that, I followed up with:

• "What is the most useful topic I could write about, considering my audience?"

In this particular case, once I had the answer to the first question, the answer to the second question was pretty much a given.

When I wrote Tempo! I discovered first-hand that writing a book is an excellent way to get in touch with people, especially if you are a bit reclusive, like me. If you write about something people care about, they will be interested, and they will help. I decided to build on that: Ask a number of managers how they have dealt with the topic I wish to cover. Write their stories. Tie back to the tools and techniques in Tempo! if and when it is useful to the reader.

With Tempo! I wrote an outline very early. This project is a bit different: I began by putting together a series of interview questions. I will go to Oslo in a couple of days, partly to discuss an outline with a friend, but I do expect that the answers I get while doing the interviews will determine the structure of the book to a large extent. Therefore I want to keep the structure at least somewhat fluid for now.

On the other hand, I want to keep the structure of the interviews the same throughout the project in order to get comparable answers from the interviewees.

I began working on the interview questions in FreeMind. FreeMind is great for jotting down ideas and structuring them. I intend to use FreeMind for the outline too. I used TextMate and Lout to write and format Tempo!, and I might stick to that combination. (I am looking for alternatives, because I want a work flow that enables me to publish ebooks in ePub format, i.e. for the iPad, with a minimum of extra work. I haven't found a good solution yet, unless I decide to bite the bullet and set up an XML authoring/publishing system.)

Thanks to a BNI referral I got a chance to make an interview with an interesting C-level executive last week. I started off by showing him all of my questions. The intent was to minimize the element of surprise. Surprises are good for birthdays, not so good when you want someone to relax and open up.

The last question I had prepared was about anonymity: May I publish the name of the interviewee? A picture? The name of the organization?

My interviewee gave me a preliminary permission to use his name, the company name, and to use a picture of him in the book. Once I have written the section about him in the book, I will mail it too him and, I hope, get a final permission.

I was careful to make the point that I would respect a desire to be anonymous. Sometimes when people are interviewed they tell a little bit more than they want the world to know. It is important to make sure people can feel safe talking to me.

I used my iPhone to record the interview. I write rather slowly, and recording the interview allows me to focus on the person I interview without worrying about missing anything important.

During the interview I tried to steer as little as possible. I asked the person I interviewed to expand on some subjects, and once or twice I made comparisons with my own experience, or some reference to a book. This worked very well. My interviewee spoke freely, and stayed on track.

Being able to stay on track is pretty important to successful C-level executives, so while i am glad it worked so well, I am not surprised.

After the interview I promised to keep in touch, and I will.

On the way back to my writing den, a café in central Gothenburg, I listened to the interview on the iPhone. Everything had worked like a charm. I had tested using the iPhone as a recorder before the interview, but it was nice to hear that I had a usable recording of the interview.

The photos I took went straight into iPhoto. I do like the facial recognition capabilities and the ability to organize photos by faces. Tempo! had over a hundred illustrations. I expect the new book to have plenty of illustrations too.

I made a write-up containing a brief biography of my interviewee, a brief list of interesting topics covered, and a summary of the interview itself. The summary has time stamps so I will be able to find the right spot in the recording easily.

I made the write-up in Pages. I normally do not want to use word processors for anything related to writing books, but in this case it might prove convenient: I intend to buy an iPad when it is released in Sweden, and use that during the interviews. I'll try to use Pages on the iPad to make the initial interview write-ups. It will be interesting to see how that works out.

In case you are wondering (though I realize you probably do not): I prefer document processors, like FrameMaker, XMetaL, and other heavy-duty tools, for writing books. I haven't found a good document processor that runs natively on my Mac though. InDesign and QuarkXpress don't cut it: No citation support. Quark does not even have ePub support yet.

In all, I couldn't have gotten a better start with the interview: I met an interesting interviewee, and I got material I believe will be very useful to the book. Better yet, I will be able to link some of it back to topics discussed and tools described in Tempo!

Next step: I have a short list of people I want to interview. I need to expand the list a bit. Some of the people I want to interview I know. There is a short list of people I want to interview because I have heard they are very good at what they do. Mostly, I will find interviewees by asking people I know whether they know a really good manager that might be interested in being interviewed. (Unless you happen to be Sir Richard Branson, Ricardo Semler, or maybe Steve Jobs, please do not tell me I ought to interview you. I want to interview leaders and managers that other people, especially people working for them, consider to be great.)

I want the book to appeal to an English-speaking audience, so I have decided to interview as many people as I can from English-speaking countries. This means I will either have to interview people via Skype (or similar), or do a lot of traveling.

Once I have enough material I will begin to organize it using tools I describe in Tempo!: Crawford Slip indexing techniques, a Current Reality Tree, and a Future Reality Tree. This will form the basis for writing the rest of the book.

I will write more about this project. If you are interested, stay tuned.

John Seddon talks about seminal moments in the development of the Vanguard method

Professor John Seddon: Seminal moments that informed the evolution of his systems thinking method from The Systems Thinking Review on Vimeo.

This is an excellent video. Hannu Kokko tweeted about it.