Friday, February 10, 2006

The Variance Trap

This posting is inspired by the matchsticks and bowls game in Eliyahu Goldratt's book The Goal.

It is hard to estimate projects. Everyone knows that. What everyone does not know, is that even if you get fairly accurate estimates for each stage in the development project, it is still possible, and quite probable, that a project will come in later than expected due to the effects of statistical variation. These effects can be proven mathematically, but it is easier, more fun, and more convincing, to do it with an experiment. Let's create a simple model of a project, and run a simulation of how variance affects development speed.

When I ran this experiment this morning, I used seven bowls that represent the stages of the project: Analysis, Design, Code, Unit Test, Integration Test, System Test, and Acceptance Test. I used colored glass beads to represent ideas that are transformed into customer valued software features in the development process.

To simulate the variance in production capabilities, I used a six sided die. The idea is to roll the die, count the eyes on the die, and move that many glass beads into the Analysis bowl. I then roll the die again, and move that number of glass beads to the Design bowl, roll again and move to the Code bowl, etc.

I can't move more beads than there is in a bowl, so if i rolled 3 the first time, and moved 3 beads into the Analysis bowl, I can't move more than 3 beads to the Design bowl, even if I roll 4, 5, or 6.

With a six sided die, the average roll is 3.5. If I roll for each of the bowls in sequence 10 times, one might expect to move about 35 glass beads from start to end. Let's call 10 such sequences one iteration.

After the first 10 sequences (80 die rolls), I had moved 21 beads through all my bowls. 21 is significantly less than 35, but on the other hand, it was the startup iteration. In the beginning the bowls are empty, so it takes awhile to get the process flowing.

On the second iteration I moved 32 beads through the process. A little bit below average, but it doesn't seem impossible to make an average of 35 over a few more iterations.

On the third iteration, I got 28 beads. A bit behind schedule, but if you were a project manager in this situation, you would probably not be unduly alarmed. (If your focus is on time reports, rather than completed features, you would probably not even know you are falling behind.)

This picture shows the bowls at the end of the third iteration:

Note how the beads have clustered together a bit in the Analysis bowl (top left). There are 3 beads in the Design bowl, nothing in the Code and Unit Test bowls, 1 bead in the Integration Test bowl, and 3 beads in the Acceptance Test bowl. One would have expected the beads to be a bit more evenly distributed, since this is a balanced system, where all parts have the same "production capacity".

The fourth iteration I moved 36 beads. The first time above average. If this was a software project, it would be taken as a sign that the work is getting up to speed. However, if you look at the picture below, you can see that there is still an uneven distribution of beads in the bowls:

There is still a cluster of beads in the Analysis bowl. There are 4 beads in the Unit Test bowl. It looks a bit funny with a group of beads in a bowl surrounded with empty bowls. It looks even more weird when you roll the die yourself, because then you can see that the beads tend to travel in waves.

At the start of each iteration, I moved the beads from the output end to the input end, so that I could reuse them. However, towards the end of the fifth iteration, I found that I was out of input beads. I had to go and borrow some extra beads from one of my wifes flower pots.

This is what the bowls looked like after five iterations. You can see that there is a new wave of beads working its way towards the end of the process:

The throughput on iteration five was 25 beads. At this point I had entered a total of 188 beads into the system. There were 46 beads in the bowls. 26 of those are in the Design bowl. The average throughput is 28.4 beads.

More than 1.6 iterations worth of beads are locked up in the system. If we had expected a throughput of 35 beads per iteration, it's well past the time to be concerned. 188 beads should not take more than 6 iterations, but it seems clear that won't happen.

I won't feed the system any more beads. Tomorrow, I'll just process the beads already there. Care to guess how long it will take to drain the system of beads and complete the project? Remember, there are 46 beads, and the average throughput rate is 28.4. Seems clear two iterations will do the job, and leave time to spare.

Well, we will see tomorrow...

See also the 2nd part of the Variance Trap entry.

No comments: