The Variance Trap, Part 4

This installment of The Variance Trap compares two similar development process simulations. They differ mainly in the amount of variance in production capability in the process stages. As the animated diagram shows, there is a great deal of difference in productivity. (If you haven't read the earlier postings on this topic, you might find that reading part 1, part 2, and part 3 makes this posting easier to understand.)
The animation above shows the result of two development project simulations. As before, the simulation model is extremely simple, with no errors or feedback loops. To simulate variations in productivity the simulation system throws a die for each process stage, for each tick of the simulation system clock.

The yellow line represents a simulation with a six-sided die. The blue line represents a three-sided die, with two added to each die roll. (A computer has no problem rolling a three sided die. If you want to do it for real, use a six-sided die, count 1-2 as 1, 3-4 as 2, and 5-6 as 3.) Lets call the six sided die 1d6 and the other one 1d3+2. (If you have ever played a roleplaying game, you wont have a problem with this notation.)

The 1d6 has a range of 1-6, and an average roll of 3.5. The 1d3+2 has a range of 3-5, and an average roll of 4. as you can see, the 1d3+2 process is much faster than the 1d6 process. If you have read the previous parts of this monologue, this should come as no surprise. The 1d3+2 process has less variance than the 1d6 process. The flow is steadier, with less inventory build up during a simulation run.

The implication is that if we can reduce the statistical fluctuations in a software development process, we can increase the productivity.

Let's take stock of what we have learned so far:

• Because of statistical fluctuations, a an unregulated development process will be slower than the slowest of the process steps. Therefore, it is impossible to accurately estimate the time required by adding together the time estimates for individual process steps. Even if the individual estimates are correct, the combined result won't be. (See Part 1 and Part 2)
  
• We can make measurements and extrapolate the time required from the aggregated process. This allows us to make fairly accurate estimates relatively early on in the project. (Part 2)
  
• The productivity will increase if the statistic fluctuations in the development process can be reduced. (Part 3)

It is time to set up a more accurate project simulation, and study the effects of different management strategies. Part 5 in this series uses a more accurate model of the development process, and explores the effects of changing the length of test, iteration, and release cycles.

Feed Me

I had forgotten to add a link to my Atom feed in the Kallokain page template. Thanks to Chris Headgate for pointing it out.

The Variance Trap, Part 3

If you read the 1st and 2nd parts of The Variance Trap, you know how statistical fluctuations can slow a project down to a velocity far below the velocity of each of the stages in the process. I demonstrated the effects of statistical fluctuations with a simple game with a die, a set of bowls that represented stages in a project, and a lot of colored glass beads. (This is a slight variation of a classic example by Eli Goldratt.)

I got tired of rolling dice by hand, so I started writing a Theory Of Constraints based process simulation engine. It is still very much a prototype, but today I wrote a computerized version of the beads and bowls game. I also managed to coax a simple animation from the simulator. It shows what happens with work buffers at different stages during the course of the project.

The simulation is extremely simplified. For one thing, there are no feedback loops from the test stages (Init = Unit Test; Intgr = Integration Test; Sys = System Test; Acc = Acceptance Test). For another, the variance is caused by a six sided die that changes the production capacity of each of the stages. There is no variance in the size of goal units (Use Cases, Story Points, or Feature Descriptions, depending on your preferences). Still, the simulation suffices to show the effects of fluctuations on a process.

The simulation run used to create the animation needed 8 (7.5) iterations to process the goal units. The average throughput per iteration was about 25. At the end of iteration 5, there were 96 goal units locked up in the process, out of a total of 188 goal units. This is a quite hefty inventory buildup. Still, it can be quite hard to even see the problem until it is too late. In many projects, work buffers are not even monitored.

At the end of the first iteration there is only a slight buildup of inventory. Seemingly nothing to worry about.

If this had been a real project, and if we had estimeted the length of it by calculating the production capacity at each stage, we would probably not suspect that anything is amiss. It is like the old story about how you can boil a live frog by putting it in cold water, and increase the temperature slowly. The frog won't notice that anything is amiss.

The image to the left is from the end of iteration 6. As you can see, there is a lot of inventory in the system. Even though Coding is done, the project is far from finished.

By this time, the frog is boiling. If we didn't know about the effects of statistical fluctuations, we might easily be led to believe that there is a problem with integration test here. There is certainly a backlog of work, but the integration test stage itself has the same capacity as the other stages.

The last goal unit leaves the system mid way through iteration 8. If this had been a real project, there would be quite a lot of frustration. What can we do? The first thing to do is to realize that there are two problems:

The first problem is how to make more reliable predictions about projects. The second problem is how to improve the development process itself.

Simulations like the one I've written about here can be a great help. In part 2, I did show a simple method of making estimates by extrapolating from the known throughput rate of the whole system. This method of estimating projects can be deduced from the simulation model. (It is not the only solution to the problem.)

The model also shows us that it is possible to improve the capacity of the system without having to increase the capacity of the capacity of the parts. How? By reducing the fluktuations. Expect a new simulation run, with the same average capacity, but less variability in the process steps, in the near future.

Until next time: good afternoon, good evening, and good night.

Mixin' Language and Culture

I've had some very positive feedback about the Extract Mixin entry awhile ago. Pat Eyler wrote a blog entry about it, and Chris Hedgate also found it useful.

In his blog, Pat asks the question whether Ruby will spawn a set of new patterns, or change the way we look at existing patterns. I firmly believe it will. It's not just Ruby though. Languages have different features, and they also emphasize different things, and have different cultures built around them.

There is a lot to learn from immersing oneself in different languages and cultures. What you learn in one place, can often be used in another. I was a Perl programmer before I began working with Java.

Perl taught me a lot about how to just get things done and that there may be more than one valid approach to solving a problem. This, I hope, fostered a spirit of openness. (Though it is really for others to say how that turned out.) It also gave me a different perspective on object oriented programming and design patterns than I got from Java. Perl also taught me the importance of unit testing, something I have brought with me ever since.

On the less technical side, the Perl community, and CPAN taught me a lot about the value of working together, and that even if you contribute only a little, you will get it back a thousand times. (Quite literally in the case of CPAN.)

One thing Java taught me is the value of a good IDE. I rarely leave home without Eclipse in my backpack (tucked away on the hard drive of my laptop, of course). Java also taught me some things about how to coordinate team efforts, and (by not having it) why it is a really good idea to have a standardized way to install libraries. I also learned not to use frameworks that have tightly coupled frameworks (anything with EJB in it), and to appreciate the power of simplicity (Spring and friends).

Java also taught me a lot about the importance of refactoring and many other development practises. Beyond that, the Java projects I worked in rekindled my interest in development methodology and management.

And now, it is Ruby. Ruby is a very nice language to work with. It allows you to cut to the chase and focus on solving a problem. The Ruby community is much like the language: friendly, open, and lazy (in the Perl sense, i.e. believing strongly in getting the maximum of value with the minimum of effort).

I like it. The Ruby community is a place where I can grow.

The Variance Trap, Part 2

This morning I continued the variance fluctuation experiment I wrote about in my previous blog entry.

I am going to show more details now, because it is interesting to follow what happens closely. (Well, it is if you are a management nerd, like me.) Remember that our original estimate, based on the average roll of the die, was that we'd get a throughput of 35 beads in an iteration. (An iteration consists of 10 sequences of 8 die rolls.) That prediction failed. The average throughput was only 28.4.

The second try to predict the end of the project used another method. I used the average flow rate, measured over the first five iterations. This prediction indicated that 1.6 more iterations would be needed. 5+1.6, rounded up, is a total of 7.

Let's see how the flow based prediction holds up. Here is state of the system two sequences into iteration 6:
The first sequence had a throughput of 0, the second had a throughput of 2. I am not feeding the system any more beads, so we can expect the group of beads in the Analysis bowl to begin to thin out. It has. There was 26 beads there, but the last two sequences have reduced that to 18. The number of beads in the Unit Test bowl (the 5th one), has 12 beads, which is one more than at the end of iteration five.

After four sequences in iteration 6, and rolling a highly unlikely series of fours and fives, sequence 3 yielded 5 beads. Sequence four yielded only 1 though, so it evens out:
The distribution continues to be rather uneven, but since there are now groups of beads closer to the end of the process chain, we can expect to make good time.

At the end of iteration 6, the model looked like this:
There are now no beads at all in Analysis, Design, Code, and Unit Test. Of course there is a weakness in the model here, because none of the test stages have a feedback loop to earlier process stages. The resulting effect is that no test ever fails. If tests did fail, that would of course slow down the process.

This is the system four sequences into iteration seven:
Turns out the flow based prediction came fairly close, predicting the end of the project 6 secuences into iteration seven. however, I made that prediction pretty late in the project. What would the flow based predictions have been earlier on? Let's look at the average flow after each iteration, and use that to calculate how many iterations we need to move 188 beads:

1. 21/1=21 ==> 188/21= 8.9
   
2. (21+32)/2= 26.5 ==> 188/26.5= 7.1
   
3. (21+32+28)/3= 27 ==> 188/27= 7.0
   
4. (21+32+28+36)/4= 29.25 ==> 188/29.5= 6.4
   
5. (21+32+28+36+25)/5= 28.4 ==> 188/28.4= 6.6
   
6. (21+32+28+36+25+31)/6= 28.83 ==> 188/28.83= 6.6

If we had watched the flow rate, we would never have underestimated, and we would have had a pretty accurate estimate after iteration 3. This suggests that monitoring the flow rate of the complete system makes it possible to make more accurate predictions than we will get by making time estimates (remember 35 beads per iteration) for each stage in the process.

In other words, measuring flow is better than estimating time!

One thing to note is that the model used here was balanced, i.e. the capacity was the same at each stage. In reality that is not the case. such differences in capacity would make traditional time estimates even more unreliable. I'll look into that, and more sophisticated methods of calculating project durationin a future blog entry. First I'll write myself a little simulation software. I'm getting tired of rolling the die.

The Variance Trap

This posting is inspired by the matchsticks and bowls game in Eliyahu Goldratt's book The Goal.

It is hard to estimate projects. Everyone knows that. What everyone does not know, is that even if you get fairly accurate estimates for each stage in the development project, it is still possible, and quite probable, that a project will come in later than expected due to the effects of statistical variation. These effects can be proven mathematically, but it is easier, more fun, and more convincing, to do it with an experiment. Let's create a simple model of a project, and run a simulation of how variance affects development speed.



When I ran this experiment this morning, I used seven bowls that represent the stages of the project: Analysis, Design, Code, Unit Test, Integration Test, System Test, and Acceptance Test. I used colored glass beads to represent ideas that are transformed into customer valued software features in the development process.

To simulate the variance in production capabilities, I used a six sided die. The idea is to roll the die, count the eyes on the die, and move that many glass beads into the Analysis bowl. I then roll the die again, and move that number of glass beads to the Design bowl, roll again and move to the Code bowl, etc.

I can't move more beads than there is in a bowl, so if i rolled 3 the first time, and moved 3 beads into the Analysis bowl, I can't move more than 3 beads to the Design bowl, even if I roll 4, 5, or 6.

With a six sided die, the average roll is 3.5. If I roll for each of the bowls in sequence 10 times, one might expect to move about 35 glass beads from start to end. Let's call 10 such sequences one iteration.

After the first 10 sequences (80 die rolls), I had moved 21 beads through all my bowls. 21 is significantly less than 35, but on the other hand, it was the startup iteration. In the beginning the bowls are empty, so it takes awhile to get the process flowing.

On the second iteration I moved 32 beads through the process. A little bit below average, but it doesn't seem impossible to make an average of 35 over a few more iterations.

On the third iteration, I got 28 beads. A bit behind schedule, but if you were a project manager in this situation, you would probably not be unduly alarmed. (If your focus is on time reports, rather than completed features, you would probably not even know you are falling behind.)

This picture shows the bowls at the end of the third iteration:



Note how the beads have clustered together a bit in the Analysis bowl (top left). There are 3 beads in the Design bowl, nothing in the Code and Unit Test bowls, 1 bead in the Integration Test bowl, and 3 beads in the Acceptance Test bowl. One would have expected the beads to be a bit more evenly distributed, since this is a balanced system, where all parts have the same "production capacity".

The fourth iteration I moved 36 beads. The first time above average. If this was a software project, it would be taken as a sign that the work is getting up to speed. However, if you look at the picture below, you can see that there is still an uneven distribution of beads in the bowls:


There is still a cluster of beads in the Analysis bowl. There are 4 beads in the Unit Test bowl. It looks a bit funny with a group of beads in a bowl surrounded with empty bowls. It looks even more weird when you roll the die yourself, because then you can see that the beads tend to travel in waves.

At the start of each iteration, I moved the beads from the output end to the input end, so that I could reuse them. However, towards the end of the fifth iteration, I found that I was out of input beads. I had to go and borrow some extra beads from one of my wifes flower pots.

This is what the bowls looked like after five iterations. You can see that there is a new wave of beads working its way towards the end of the process:


The throughput on iteration five was 25 beads. At this point I had entered a total of 188 beads into the system. There were 46 beads in the bowls. 26 of those are in the Design bowl. The average throughput is 28.4 beads.

More than 1.6 iterations worth of beads are locked up in the system. If we had expected a throughput of 35 beads per iteration, it's well past the time to be concerned. 188 beads should not take more than 6 iterations, but it seems clear that won't happen.

I won't feed the system any more beads. Tomorrow, I'll just process the beads already there. Care to guess how long it will take to drain the system of beads and complete the project? Remember, there are 46 beads, and the average throughput rate is 28.4. Seems clear two iterations will do the job, and leave time to spare.

Well, we will see tomorrow...

See also the 2nd part of the Variance Trap entry.

Skunkworks Project: Apple's Graphing Calculator

Here is the story of a famous skunkworks project at Apple, the Graphing Calculator. It provides fascinating insights into how and why software developers work. Also, it is a fun story.

My favourite quote from the story is:

"I asked my friend Greg Robbins to help me. His contract in another division at Apple had just ended, so he told his manager that he would start reporting to me. She didn't ask who I was and let him keep his office and badge. In turn, I told people that I was reporting to him. Since that left no managers in the loop, we had no meetings and could be extremely productive."

Arguably, Ron Avitzur, who wrote the story, was in fact the project manager, but he certainly did not think of himself that way. It is unlikely that anyone else did either. He just did what had to be done to make the project work.

Sounds to me like Alistair Cockburn's "ri" stage of project management.

New Theory of Constraints Link

I added one more entry to my gaggle of links. It is a very good Theory of Constraints page.

My Brain Is Constipated

Working...on...Theory Of Constraints essay.

So tired...

Need...more...brains...

---

Actually, it isn't that bad. I have spent a large portion of this week, and the last, working on an essay that summarises the basics of TOC and Throughput Accounting. I have worked with agile methodologies for several years, and have now reach a point where it is time to stop, reflect on what I know, and check if the theory and my experiences match. So far they do, which is very nice.

Management theory may sound boring, but it isn't. Right now I am focusing on the mathematical side of management theory, and it is fascinating how an equation can leap out at you and say things like "move those desks closer together, and that screen from here to there", or "if you move one person from analysis to test, you will make the deadline without overtime".

Most managers, including me, go by experience, best practises and rule of thumb most of the time. This is a good thing. It makes for quick, flexible decision-making. On the other hand, if you don't verify your practises and rules against both reality and the models describing them, how can you know if you are doing things right? How do you know if you are doing the right things?

For example, most managers do know that iterative development is better than waterfall development. They also know that short iterations are better than long ones, as long as they aren't too short.

Exactly why is it that iterations are better than waterfall? Most would answer "better steering", and be satisfied. That is only part of the answer. The rest of the answer has to do with reduced cycle times, and increased speed of development. If you reduce batch sizes (iterations, design-test-code cycles, delivery cycles, whatever), the process will go faster. This isn't intuitively obvious, and it is nice to be able to prove it mathematically.

I'll publish a link to the essay when it is finished

Ten Rules for Web Startups

I found ten rules for web startups that made a lot of sense.

The Customer Drives the Car

I talked with a friend, who is a very, very good programmer, about why it is so hard to find sane management practises in software development projects. Most project managers I meet (with a few shining exceptions) know very little about good management practises and process control.

My friend mentioned that IBM is moving to fixed price contracts for most of its subcontractors, and said he hopes other customers will follow. He pointed out that with time and materials contracts, there is no incentive for custom software development companies to improve their practises.

I believe he is right. I am no friend of fixed price contracts. Most projects have considerable feature creep, not just because of sloppiness, but because new requirements are discovered during the development process. There may be no way for the customer to know at the outset of the project that a particular feature will be needed. For this reason, I am more in favor of other contract types, like staggered contracts, profit sharing, target cost, and target schedule contracts. Generally speaking, all of these are more suited to software development than time and materials or fixed price contracts. (All of these feature negotiable scope, which is a key to success when developing software.)

Still, fixed price contracts do provide an incentive to do the work in a professional manner, so they would force the industry to shape up a bit. There are also ways to reduce the risk of fixed price contracts.

The important thing here is that the incentive to change must come from those who have power and something to gain from a change, and that is the customers. This thing has happened before, in the automotive industry. When Toyota went Lean in the late forties-early fifties, they simply refused to make business with subcontractors, unless they to went Lean.

The thing that made this actually work, was that Toyota sent their own Lean experts to their subcontractors, and helped them reorganize their companies and their manufacturing processes. It is as Kent Beck wrote, "the customer drives the car".

The nice thing about this, is that the subcontractors ended up more profitable than before. The scary thing is that they resisted becoming more profitable until they were forced by their main customer. Even scarier, many software development companies are doing that now.

On the other hand, some software companies are getting it. Microsoft has had great success implementing Drum-Buffer-Rope in an IT department in India. (And here is a presentation slide version.)

Now, if I could find a software development company interested in doing the same kind of thing where I live... (If you just happen to work at such a company, why not email me.)