Kallokain

Lately, I have had reason to study the effects that Design-In-Process, DIP, has on the profitability of software projects. It turns out DIP is a project killer, and potentially a corporate business killer too. What is Design-In-Process? It is partially finished work. Most software development projects have too much of it. The manufacturing industry learned decades ago that too much inventory was a bad thing. In the software development business, the inventory is Design-In-Process. We still haven't learned the lesson. Why is DIP bad? Let's have a look at the classic Return On Investment (ROI) equation: ROI = (T - OE) / I Where: ROI is the Return On Investment, a measure of how profitable a business system (company, project, etc.) is T is Throughput, the amount of money made OE is Operating Expenses, the money the business system spends producing throughput. Wages, rent, electricity, are all OE. I is investment, the money tied up in the system. This includes computers...

While looking for material about queue theory, I came across a traffic simulator that is worth a look. With it, you can simulate how traffic moves, and doesn't move, in different kinds of situations. It is worth a look.

I have begun working in a new project recently. It is the first Scrum project at Wireless Car, and they hired Tobias Fors from Citerus to get us started. Tobias presented Scrum in a manner that was both entertaining and informative. The kick-off took two days. The first day was a Scrum presentation, the second day we did Sprint planning. It is rare to meet someone that can talk about the same subject matter for two days straight without being boring even once. I usually make people's eyes glaze over within a few minutes, so I was very impressed with Tobias. Also, for two days I had someone new to talk about agile and TOC and queue theory with. Great!

I missed the first Gothenburg Ruby User Group (GRUG) meeting, but not the second, last Thursaday evening. 15 geeks (only a few of whom are shown in the picture), and Ruby. Doesn't get much better. Niclas Nilsson and Karl-Johan Kihlbom talked about the recent RailsConf conference in London. The main event, arranged by Emily Bache, was a code kata. You can read more about it in the GRUG Google Group at http://groups.google.com/group/grug . The next meeting will be in a month or so. I'm looking forward to it already.

I have written a presentation comparing flow organizations and functional organizations. You can check it out here if you are interested in this sort of thing.

In the Systems Thinking series of posts (also including Systems Archetype: Shifting the Burden and Systems Archetype: Shifting the Burden to the Intervenor ), I try to show the basics of how complex business systems interact. With the systems archetypes, it is possible to find standard solutions to common problems, but there is still a piece missing. In a specific case, how do you find the root cause to your problem? Have you ever worked in a project, or workplace, where it feels as if you are spending all your time fighting little fires? Usually, all those little fires stem from a few root causes. Unless you want to spend your time fire-fighting, you had better find the root cause of your problems, and deal with that. There are many methods of finding root causes. One of my favorites, because it is so simple, is Toyota's Five-Why method. The idea is that when you are faced with a problem, you ask why that problem occurred. When you find the thing that caused your problem, you as...

So, maybe it's not a full month. Then again, I'm not really back. I am writing this in an Internet cafe in Ho Chi Minh City, Vietnam. If you want to study one-piece-flow, and want to see why moving small batches around is better than lugging large ones, HCMC is the perfect place. The streets here have more traffic than anything I've ever seen. There are very few traffic lights, and driving on the right side of the road is sort of a very loose agreement. Also, the most common vehicles are scooters and light motorcycles. There are 30 million scooters in Vietnam, and by the look of it, most of them drive by my hotel each day. (I'll upload some pictures when I get back.) One would expect the result to be total chaos and confusion. It isn't! Actually, the traffic flows like nothing you've ever seen (unless you've been here, of course). There are two reasons for this. The first is that though the streets are packed with vehicles, each vehicle is small and manouver...

I'm taking a month off from blogging. Will be back in September.

There is a special case of Shifting the Burden which I am especially familiar with because I have worked as a consultant for many years: Shifting the Burden to the Intervenor . Description The problem may occur when an organization decides to rely on outside competence to solve an internal problem. The problem may be solved. The catch is that the competence to deal with the internal problem now resides outside the organization. This makes the organization vulnerable if the problem occurs again. "The organization" may be a company that outsources development work to a consultant, or outsources a part of their infrastructure to a subcontractor. Departments within an organization are also vulnerable. They may (not always voluntarily) let another part of the organization take over an important function, only to find that the other part of the organization is not able to process requests quickly (usually because of queues), may not fully understand the requests (development depar...

Description Shifting the Burden is one of the more common systems archetypes. The problem occurs when a short term fix is applied without regards for possible side effects. The short term fix seems to work, but over time the side effects escalate. In many cases, the capability to apply a long term solution atrophy over time, making it very hard to correct the problem. Example A classic Shifting the Burden example in software development is when management tries to apply a brute force solution, like increasing the manpower of a project, or forcing developers to work an excessive amount of overtime, to compensate for low productivity, or poor planning. The diagram above describes a situation where a project has productivity problems. In other words, it won't make the deadline. The obvious, but often flawed, solution is to add more people to the project. Most projects measure progress in terms of hours worked. Unless you live on Mars, you are almost certainly familiar with the basic ...

Systems thinking is a theory that argues that social systems must be studied as a whole. It is impossible to predict the effects of the whole system by studying its parts in isolation. A "social system" can be anything from a family unit, to a system of interacting economic units (companies, countries, etc.), or even an ecological system. Systems thinking is an important part of the theoretical foundation for agile software development methodologies. I got interested in it some years ago because I kept stumbling on references to the systems thinking bible, The Fifth Discipline , by Peter M. Senge, in books about agile methodologies. Somehow, there was always one more programming book to buy first, but then I spied a copy at my parents-in-law's house, and borrowed it. After a few pages, I was hooked. Since then, I have worked systems thinking into the way I work. I have found it to be a valuable tool when trying to figure out what is really happening in different situation...

A while ago (in another company than the one I work for now) I worked for a company that did not believe in making things easy for their developers. Everyone had to work in an open landscape. The landscape was divided into rectangular cells. Developers sat at corner desks, so that developers in a cell faced away from each other. To make communication between developers even more difficult, people working on the same project were usually located in different cells. A new and very complicated project began. We developers realized that the seating arrangements would never work. There was no way we could succeed if we could not talk to each other. We decided to ask our department manager for a project room of our own. The department manager realized that we needed to work close by to have the slightest chance of success. "OK," he said, "you'll get the room, for this project, because it is an unusually difficult one, but of course you can't get one every time you need...

A couple of people have emailed me about missing images and animations in my Variance Trap series of postings. The broken links were due to a reorganization of my web site. I have finally fixed the problem. You can view the postings here: Part 1 , Part 2 , Part 3 , Part 4 , Part 5 .

Apress has listed a new Ruby book, Pro Ruby , among its forthcoming titles. Pro Ruby shows how to combine Ruby and agile methodologies into a unified, powerful package. I do have a special interest in this book, because I'm writing it. (Which is why I have blogged less than usual lately.) I don't want to give anything away prematurely, but I can tell you two things about the book: It does not have a Hello World example There is a sample application, but it isn't an online shopping application There will be some time before Pro Ruby hits the book stores. When it does, I hope you like it.

This site is too good to miss: http://thedailywtf.com/

You know all about the Agile Manifesto , of course, but what is the Declaration of Interdependence ? It is the next logical step, the answer to the question how Agile principles can be extended to non-software projects, and management in general. Alistair Cockburn has written about it on his site , and in an article for Better Software Magazine. The declaration is a year and a half old, but the article was published in June. Do read the article. It does show how profoundly different an Agile business is from the ordinary kind. (Technically, "the ordinary kind" is a business based on the principles of Scientific Management and cost accounting .) Just a little something to show that the blog is alive. Have been quite busy the past few weeks, but will make time for more blogging. (I am not sure if that is a threat or a promise.)

I had some good news a couple of days ago, and I just can't resist toting my own horn a little. A couple of years ago I was subcontracted to write a web based print formatting system. The system could automatically produce brochures, and used FrameMaker as a formatting engine. I haven't counted this system as one of the great successes of my career. FrameMaker and Internet Information server did not play well together. (Neither did Apache and FrameMaker.) In the end, I solved the problem by writing a small web server of my own. Though the solution worked, my client was not happy with the design. He insisted he wanted the application to run under IIS, even though it was not technically possible. Fortunately my client's client did accept the solution. Still, my client wasn't happy, so I wasn't happy. Earlier this week a friend of mine told me what became of my 'failed' system. The system has worked very well for more than five years now. The customer estimates...

A friend and I spent an enjoyable afternoon debugging a Rails application yesterday. "Enjoyable" and "debugging" usually don't go together, but in this case they did. My friend, who happens to be a much better programmer than I am, was stuck on a bug. His application had a form that worked correctly the first time it was used. The second time around, values filled in the first time reappeared in the form. In the end it turned out that an automatically generated find_xxx_by_id method returned the wrong data the second time it was called. (My friend is filing a bug report.) Our excursion into bug country was enjoyable because we solved the problem as a team. It made it easier to keep focused. There was communication. We could bounce ideas off each other. I learned a few new things about ActiveRecord. We solved the problem, and the bug report may save other people some trouble in the future. It was a good half days work.

The connection to www.henrikmartensson.org is incredibly slow at the moment. Don't know why. My site host and I will look into it.

Avoid method name collisions by mixing in modules to an instance of an inner class. class Outer class Inner end def initialize(mixin) @inner = Inner.new() @inner.extend(mixin) end ... end When extending objects with mixin modules, there is a risk of method name collisions. Given the example in the Module Injection post , the execute_before and execute_after methods in the Strategy object might collide with method names in a rule module. Suppose the rule module looks like this: module TransformationRules def functiondef_before(element) ... end def functiondef_after(element) ... end def execute_before(element) ... end def execute_after(element) ... end end When the dispatch mechanism in the Strategy object finds an execute element, it will dispatch the element to the wrong execute method. It will call the execute_before method defined in Strategy, not the one defined in the TransformationRules module. Pollution Control reduces the ...

Signal that objects of a class is designed to be extended via mixins by passing the mixin module via the constructor. class ExtensibleClass def initialize(mixin) extend(mixin) end ... end Ruby allows objects to be extended with new functionality at runtime. Any object can be extended in this manner by a call to the extend method. Some classes are designed to have their objects extended with new functionality. Example Consider a Strategy object that encapsulates a set of transformation rules for XML documents. The document is used by a treewalker. The treewalker walks the node tree representing the XML document. The Strategy class uses Module Injection: strategy = Strategy.new(io, TransformationRules) treewalker = TreeWalker.new(strategy) treewalker.walk(xml_document) The strategy object contains dispatch code that chooses which transformation rules to call depending on such things as node type (element, processing instruction, text, etc.) and node name. The Strategy object ...

My publisher (Apress) is looking for an author interested in writing an advanced book on Rails. They work with new authors all of the time, so prior writing experience isn't necessary. If you're interested send myeditor (Jason Gilmore) and email at jason@apress.com .

I attended a talk by Lars Taxén about Integration Driven Development and Anatomies today. IDDA is a development methodology for large projects. "Large" means 1,000+ people. IDDA was developed at Ericsson for use in telecommunications projects. One of the things I found interesting was that the anatomies part, a method for modelling complex systems, is quite similar to Domain Modelling. That is, both methods aim to create models that mirror real world objects or processes. Anatomies, like Domain Models, make use of bounded contexts to separate different parts of the system. Some things were very different from what I am used to. For example, IDDA subsystems are implemented in the order they are activated when the system is used. That is very different from the Agile practice of implementing the most valuable functionality first. For example, consider a document production system consisting of three subsystems: an XML authoring system, a Document Management System (DMS), and a...

Finally, I have put www.henrikmartensson.org on Rails! I have rebuilt the site from the ground up with Ruby on Rails . It was a pleasure to do it. The visible changes are small. There are a few more menu choices, including a link to this blog. The site also lists the most recent postings to this blog. Behind the scenes, the site is now database driven. Articles, presentations and images are stored in a database and retrieved on the fly. This makes the site a lot easier for me to update and maintain. I hope this will encourage me to write more articles. We'll see what happens. I had an unexpected problem with reading the Atom feed from this blog. I tried two different Atom modules, but neither could parse the feed properly. Both had trouble with Atom entries that contained more than one link. The problem was easily solved with REXML and XPath. The current implementation is a bit of a hack, but I'll improve it in the near future. I think there is a blog post just waiting to be w...

Pascal Van Cauwenberghe at Thinking For A Change has made the kind of blog post I like the most: it's about me . Well almost, it is about the Variance Trap series and the Theory Of Constraints based software development simulations I have run. His post also reminds me that I haven't quite finished the series yet. One more article to go. Coming real soon now...

Paul Battley emailed me about a bug in Test::Unit::XML . Attribute strings containing entity references were reported as not being equal even if the strings were exactly alike. I fixed the bug, added an assert_xml_not_equal assertion for good measure, and made a new release. The assert_xml_not_equal assertion is the inverse of assert_xml_equal . It is mostly a convenience method. That is, it was convenient for me to have it when I wrote a test that compared attribute values. I haven't worked much on my open source projects lately. It is not that I have lost interest. It is just that I have been otherwise occupied. Thus I am doing defect-driven development at the moment. When someone finds a bug, I fix it. If I can make some small improvement without spending a lot of time on it, I do that too.

Joel Spolsky has written an interesting article about how to organize and run software companies. Here is a quote: The command-hierarchy system of management has been tried, and it seemed to work for a while in the 1920s, competing against peddlers pushing carts, but it's not good enough for the 21st century. For software companies, you need to use a different model. And here is another: ...if only 20% of your staff is programmers, and you can save 50% on salary by outsourcing programmers to India, well, how much of a competitive advantage are you really going to get out of that 10% savings? I think you will find the article an interesting read.

At Google, they have an 80/20 rule . Developers spend 80% of their time on work that has been assigned to them, and 20% is their own time, to spend on projects that are just interesting . That is one day per week that a Google employee can sit thinking things through, dreaming new stuff up, and implementing it. Does it work? AdWords was created this way , and it is a major revenue maker. Google employees seem pretty enthusiastic. Google does attribute part of its success to the system. It seems to work very well, at Google. Of course, at Google they have a pretty open and enthusiastic attitude to new ideas. "When someone comes up with a new idea, the most common response is excitement and a brainstorming session" according to a post by Joe Beda at Google. According to Beda, managers at Google actively encourage employees to come up with new ideas. It is something to think about.

Pascal Van Cauwenberghe has posted an interesting article on the danger of succeeding with Theory Of Constraints. He is quite right. If your part of a process has been the system constraint, and you elevate it, some other part of the process will be the new constraint. The people responsible for that part of the process may be rather unhappy with you. A friend of mine took a job as a subcontractor. He finished his job ahead of time, with no defects at all in his work. His manager was not happy. My friend was accused of "idling", because he had finished ahead of schedule. The idea that it was the manager who was slow in obtaining more work was of course never under consideration. Nor was it ever under discussion that if the contract had been anything other than a time-and-materials contract, finishing early would not have reduced the net profit. Quite often the slowpokes protect each other without even realizing they are doing it. I once worked at a company where nothing ha...

I once worked in a Java project where cross-cutting concerns where a big issue. One of the problems was logging. Error handling was another. Logging and error handling code was sprinkled throughout 15 MB source code. Fixing it was no picnic. Today, I once again had reason to reflect on how to separate cross-cutting concerns. This time in Ruby. For example, if I have a Ruby class like this: class TemperatureConverter def celsius_to_fahrenheit(c) 9.0 / 5 * c + 32 end def fahrenheit_to_celsius(f) 5.0 / 9 * (f - 32) end end it would be very nice if I could add a cross-cutting concern without having to modify the original code. I would like make the methods in the class loggable by opening the class and declaring that I want logging, like this: class TemperatureConverter loggable_method :celsius_to_fahrenheit, :debug loggable_method :fahrenheit_to_celsius, :debug end How do I do that? I can add loggable_method as an instance method to Object . Object is the parent class of all other cla...

Noel Llopis who runs the Games from Within blog has posted an interesting article about a day in the life at High Moon Studios , the Agile games company were he works. High Moon Studios uses Scrum and Extreme Programming. Judging from the article, they have got it down very well.

Scott Klebe asked me a very interesting question in a comment to my previous posting: does my research show whether other planning techniques are effective in increasing predictability? Here is my reply . The short version of the reply is: yes, I believe some methods are effective, but if there is a lot of unpredictability, it is better to just track what happens and then extrapolate from the results. The catch is that the variance that causes unpredictability does not necessarily arise from within the project. That makes the effects hard to predict, and difficult to control. Expounding on this a bit further: in The Variance Trap series I have modeled projects as systems standing in isolation. This is a huge simplification. In real life, there is a lot of interaction between the project system, and other systems. A typical development team, consisting of a number of consultants (that's typical in my line of work) will be influenced by: Groups within the teams own organisation, ...

I have just set up a TocSim home page at http://tocsim.rubyforge.org/ , and made TocSim available on RubyForge. There is no gem package or tarball available yet, so anyone interested will have to download directly from the repository.

Sean P. Goggins has written an interesting blog entry about Orchestrating Enterprise Architecture . If you read it, spend a little extra time thinking of the idea of single and double loop learning. It is worth it. Do you work in an organization where double loop learning is common? If so, why not drop me a line? I'd like to know more about how your organization works.

This is the fifth part in an ongoing series of articles. You might wish to read part 1 , part 2 , part 3 , and part 4 before reading this one. Until now every simulation I have described has used an extremely simplified model of a development process. It is time to try a more accurate model, and see where that gets us. Here is the model I am going to use: (Click on the image to see a full scale version.) The numbers in the boxes at each stage in the process indicate how many goal units that can be produced in a single day. To this number, I add a random variation of +/- 2 units to represent fluktuations in production capacity. (Judging from experience, this is a lot less than in many projects. I want to err on the side of caution. 50% of all goal units fail at Unit Test. The effort needed to fix a defect at unit test is 10% of the original effort. Integration tests have a 20% failure rate, and the effort to fix is 10% of original effort. System tests have a 10% failure rate, but since...

I have poured all my spare time in TocSim, the Theory Of Constraints simulator, lately. Things are going well, but I have to constantly fight creeping featuritis. Declan , another of my projects, hasn't seen much development lately, but that will change. I am going to present it at XTECH 2006 in may. If you are going to the same conference, why not drop me a line .

This installment of The Variance Trap compares two similar development process simulations. They differ mainly in the amount of variance in production capability in the process stages. As the animated diagram shows, there is a great deal of difference in productivity. (If you haven't read the earlier postings on this topic, you might find that reading part 1 , part 2 , and part 3 makes this posting easier to understand.) The animation above shows the result of two development project simulations. As before, the simulation model is extremely simple, with no errors or feedback loops. To simulate variations in productivity the simulation system throws a die for each process stage, for each tick of the simulation system clock. The yellow line represents a simulation with a six-sided die. The blue line represents a three-sided die, with two added to each die roll. (A computer has no problem rolling a three sided die. If you want to do it for real, use a six-sided die, count 1-2 as 1, 3...

I had forgotten to add a link to my Atom feed in the Kallokain page template. Thanks to Chris Headgate for pointing it out.

If you read the 1st and 2nd parts of The Variance Trap, you know how statistical fluctuations can slow a project down to a velocity far below the velocity of each of the stages in the process. I demonstrated the effects of statistical fluctuations with a simple game with a die, a set of bowls that represented stages in a project, and a lot of colored glass beads. (This is a slight variation of a classic example by Eli Goldratt.) I got tired of rolling dice by hand, so I started writing a Theory Of Constraints based process simulation engine. It is still very much a prototype, but today I wrote a computerized version of the beads and bowls game. I also managed to coax a simple animation from the simulator. It shows what happens with work buffers at different stages during the course of the project. The simulation is extremely simplified. For one thing, there are no feedback loops from the test stages (Init = Unit Test; Intgr = Integration Test; Sys = System Test; Acc = Acceptance Te...

I've had some very positive feedback about the Extract Mixin entry awhile ago. Pat Eyler wrote a blog entry about it, and Chris Hedgate also found it useful. In his blog, Pat asks the question whether Ruby will spawn a set of new patterns, or change the way we look at existing patterns. I firmly believe it will. It's not just Ruby though. Languages have different features, and they also emphasize different things, and have different cultures built around them. There is a lot to learn from immersing oneself in different languages and cultures. What you learn in one place, can often be used in another. I was a Perl programmer before I began working with Java. Perl taught me a lot about how to just get things done and that there may be more than one valid approach to solving a problem. This, I hope, fostered a spirit of openness. (Though it is really for others to say how that turned out.) It also gave me a different perspective on object oriented programming and design patt...

This morning I continued the variance fluctuation experiment I wrote about in my previous blog entry . I am going to show more details now, because it is interesting to follow what happens closely. (Well, it is if you are a management nerd, like me.) Remember that our original estimate, based on the average roll of the die, was that we'd get a throughput of 35 beads in an iteration. (An iteration consists of 10 sequences of 8 die rolls.) That prediction failed. The average throughput was only 28.4. The second try to predict the end of the project used another method. I used the average flow rate, measured over the first five iterations. This prediction indicated that 1.6 more iterations would be needed. 5+1.6, rounded up, is a total of 7. Let's see how the flow based prediction holds up. Here is state of the system two sequences into iteration 6: The first sequence had a throughput of 0, the second had a throughput of 2. I am not feeding the system any more beads, so we can exp...

This posting is inspired by the matchsticks and bowls game in Eliyahu Goldratt's book The Goal . It is hard to estimate projects. Everyone knows that. What everyone does not know, is that even if you get fairly accurate estimates for each stage in the development project, it is still possible, and quite probable, that a project will come in later than expected due to the effects of statistical variation. These effects can be proven mathematically, but it is easier, more fun, and more convincing, to do it with an experiment. Let's create a simple model of a project, and run a simulation of how variance affects development speed. When I ran this experiment this morning, I used seven bowls that represent the stages of the project: Analysis, Design, Code, Unit Test, Integration Test, System Test, and Acceptance Test. I used colored glass beads to represent ideas that are transformed into customer valued software features in the development process. To simulate the variance in produ...