Quote Of the Day

Culture is not a product of what management says, it is the result of what management does.
- Bill Waddel in the Evolving Excellence blog, 24th of January 2006

Waterfall 2006 Conference

If you are bored with all the new fangled Agile stuff, why not try a more traditional approach to software development?

Refactoring: Extract Mixin

I think I might have a previously uncatalogued refactoring here. This is an alpha version. If at least a few people find it useful, I'll do a proper writeup. Here goes:

You have two classes or mixins with similar features.

Create a mixin module, and move the common features to the mixin.

This refactoring is applicable with languages that support mixins, like Ruby.

Motivation

The primary purpose of Extract Mixin is to eliminate code duplication. In this respect, this refactoring is very similar to Extract Superclass. There are some noteworthy differences:

• Mixins are not a part of the inheritance hierarchy. Thus, it is possible to mix any number of mixins into a class, or into other mixins. Consequently, mixins are more flexible than class hierarchies.
• Mixins can create instance variables, but this introduces the risk of naming conflicts. Therefore, this refactoring is safest with methods that get all their variables as parameters to the method call.

Extract Mixin can also be used to extract, modularize and encapsulate cross-cutting concerns, for example logging and authentication, in a manner similar to applying advice in Aspect-Oriented Programming (AOP).

Mechanics

1. Run the test suite to ensure that everything works as it should.
   
2. Create a new file containing an empty mixin module.
3. Include the mixin in the original class (or original mixin).
4. Run the tests
5. Move methods, one by one, to the mixin module. Run the tests after each move.
6. Examine the methods left in the class(es). If there are common parts, use Extract Method to extract those parts into separate methods, then move the methods to the mixin. Run the test suite to verify that everything works.
   

Example

The following two classes both have a walk() method, but they are in different inheritance hierarchies:


class Human << Mammal
def walk()
...
end
end

class Penguin << Bird
def walk()
...
end
end


By factoring out the walk() method into a mixin, we can eliminate the code duplication:


module Biped
def walk()
...
end
end

class Human << Mammal
include Biped
end

class Penguin << Bird
include Biped
end

Ruby Refactorings

I think I accidentally discovered two new refactorings today, Extract Mixin and Replace Mixin With Class. (I'll make detailed writeups in a day or two.) This morning I found myself in central Gothenburg with nothing to do for an hour before a meeting, so I ducked into a cafe, had a cup of tea, and began refactoring Test::Unit::XML.

If you have had a look at the code in the assertions mixin, xml_assertions.rb, in version 0.1.4, you'll notice that it is fairly heavyweight, weighing in at more than 220 lines of code. This is a fairly large chunk of code, and all of it is for a single assertion, assert_xml_equal. A big part of that code is for comparing different types of REXML nodes, REXML::Element, REXML::Text, and so on.

The node comparison code was split into fairly small methods, so it wasn't too shabby, it was just that it was located in the wrong place. It made sense to move that code to a class of its own.

I did consider moving it to a mixin module, but in the end I decided against it. The main reason is that the XML assertions in Test::Unit::XML are mixed in with the Test::Unit::TestCase class. I don't want to pollute Test::Unit::TestCase with a lot of methods that don't need to be there.

However, extracting the comparison methods into a mixin of its own was a logical first step. I created a mixin module, moved the XML comparison methods there, added an include statement to the xml_assertions.rb file, and ran my test suite. Everything worked.

The second step was to replace the new comparison mixin with a class. First, I wrote a comparison class, and included the comparison mixin. Second, I changed the XML assertion module in xml_assertions.rb so that it instantiated the comparison class and called its methods instead of using the comparison mixins directly. Third, I ran the tests, and they worked. Fourth, I removed the include statement that included the comparison mixins in the assertions module. Step five was to move the comparison methods into the comparison class and remove the comparison mixin. Finally, I ran the tests again, and everything was OK.

People Are Downloading Test::Unit::XML

I just checked some Test::Unit::XML download statistics over at RubyForge. Looks like 15 people have downloaded the package so far. (The chart says 18, but I have made three test downloads myself, so they don't count.)

Due to the smashing success, I am going to treat myself to a bit more Ruby programming tomorrow.

Test::Unit::XML 0.1.4 Update

Time for a development diary entry.

I've just released Test::Unit::XML version 0.1.4, an XML test framework for the Ruby programming language. This release adds doctype comparisons to the assert_xml_equal assertion. There are also improvements to the documentation, and minor updates to the tutorial.

Barring bugs, assert_xml_equal is now finished. The next release will feature at one more assertion. I'm keeping the releases really small in order to provide new features as frequently as possible.)

Internally, the structure of the framework will change substantially. In version 0.1.4, the bulk of the functionality is in the xml_assertions.rb file. This includes a truck load of methods for comparing various elements. These will be factored out into a separate class. From release 0.1.5 onwards, xml_assertions.rb will delegate as much work as possible.

Version 0.1.4 has 17 tests, with a total of 67 assertions. That's a lot of tests for testing a single assertion. Still, I've had to comment out four assertions in two tests, because I hit a snag with the REXML parser when parsing notations. Under some (uncommon) conditions, the parser may miss the end of a notation declaration, with unpredictable results, and without reporting an error. I have reported the bug, so with luck, it may be fixed in the next release of REXML. Until then, I will keep the aforementioned assertions commented out.

The problem isn't serious. Apparently, the problem is so uncommon it doesn't even show up in the W3C test suite for XML parsers. (REXML handles all of those.)

Watch out for version 0.1.5 in a week or two.

Project Management Patterns

I had a thought while doing a bit of editing on my previous blog entry. One thing that could be useful to software development projects is a set of patterns for project managers, like design patterns for programmers, or system archetypes for line managers. Most development methodologies have plenty of artifacts, but no management patterns.

It may be that the systems archetype pattern language from Systems Thinking is a good fit for this.

I need to cogitate a bit on this, and try a few things out, but I will return to the topic.

On Project Economics

I visited the easyFair convention at the Swedish Exhibition Centre in Gothenburg today. I found three different vendors who sold project management software. All three products had functions for tracking the cost of a project. This isn't surprising of course. What I found interesting is that all three products used the same economic model to do it, and that model is wrong.

Huh? How can that be possible? It is not as strange as it sounds. All three products had functions for tracking the hours worked by each project member, multiply it with the hourly cost, and sum the costs for all project members, like this:

project cost = Σ(cost/hour * hours)

A simple formula, but there is something missing. Try this one instead:

project cost = Σ(cost/hour * hours) - profit

The profit here is what the customer earns from getting frequent partial deliveries during the course of the project. (It's not the total profit from the project. Oh, how I wish for MathML support in more browsers.)

In a waterfall style project, with a single delivery at the end of the project, the profit will be 0, because the software won't be used until the project is over. On the other hand, in an iterative project with frequent deliveries, such as an XP, Scrum, or FDD project, the profit may be substantial. In some cases the profit is higher than the cost of development, which means a project pays for itself even before it is finished. (I won't go into the details of how to do a Cost/Benefit analysis here.)

The problem is that by focusing only on the costs, a project manager, and other stakeholders, get a flawed view of the economic viability of a project. This is bad because it is economics that drive project decisions.

Should we add more people? Should the project be cancelled? Should we reduce the project staff? Where should we focus our efforts? The decisions are all heavily influenced by the economic model of the project.

For example, if a project can make a delivery every two months, it may be worth while, but if the deliveries are every four months, it may not be. With this information at hand, a project manager can take measures to shorten the delivery cycle, for example:

• Reduce the size of use cases (for example by splitting large ones into smaller ones) to reduce batch sizes.
• Make sure to use developers that know how to write unit tests, to ensure that the software is more stable during development.
• Use automated builds and testing.
• Prioritize features so that the most important ones (from the customer's perspective) are tackled during the first iterations.
• Reduce iteration length to get better control of the state of the system.

etc., etc.

With a flawed economic model as the basis for decisions, the wrong decisions will be made. If a flawed model is built into the management tools, it becomes very hard to detect it. Most project managers I know just trust the tools.

The Lean Link

I just added a link to Poppendieck.LLC. This is the web site of the famous management experts Tom and Mary Poppendieck. Their book Lean Software Development is a must for anyone, customer, manager or developer, seriously interested in software development. There is a sample chapter here.

The CHAOS Ten - and the Missing Item

I had a look at the Standish Group's CHAOS report of 2001. The report contains an analysis of the factors that contribute most to project success. In order of importance, the factors are:

1. Executive Support
2. User Involvement
3. Experienced Project Manager
4. Clear Business Objectives
5. Minimized Scope
6. Standard Software Infrastructure
7. Firm Basic Requirements
8. Formal Methodology
9. Reliable Estimates
10. Other

According to the report, the top item, Executive Support, is about 3 times as important as the number eigth item, Formal Methodology.

When I hold talks about software development, which I do a bit sporadically these days, I am most interested in talking about the top five items on the list. Not because of the report, BTW, but because my own experiences, and those of others, have led me to believe that these factors are important.

I also like to talk a bit about number eight, Formal Methodologies. This is because of a personal belief that methodology matters a bit more than the list implies. Why is that? It is because the methodology guides how we think about the other items on the list.

Sometimes, having a formal methodology becomes a substitute for thinking. If we do what the methodology says we should do, we won't get blamed, even if we fail. Well, there may be a bit of yelling, but probably few consequences beyond that.

On the other hand, not following the methodology is dangerous. If we fail, there may be severe consequences. There may even be unfortunate consequences if we succeed, because in many corporate cultures, following the rules is more important than succeeding.

Which brings me to the missing item on the CHAOS list: Corporate Culture. That is where everything begins. The corporate culture has a strong influence on all the items on the list.

A project team can get executive support even if its not a part of the culture. However, the team is much more likely to get such support if it is part of the culture. Also, such support is much more likely to matter. Same thing with user involvement. If its not part of the culture, for both vendor and client, user involvement is likely to dwindle over time, even if things start out well.

Going down the list, corporate culture is a strong, but often unnoticed, influence on everything. I believe it should be the top item. Therefore, it is also my favorite subject for talks. It is also why my articles at www.henrikmartensson.org tend to be about corporate culture and management, even though I am a developer at heart.

Stephen Chu's Agile Story Tips

Stephen Chu (whom I don't know) has some interesting tips on writing Stories for agile software development. I found his posting on ThoughtBlogs, and it prompted me to have a look at Chu's blog. Interesting stuff on .NET, Ruby, and development in general.

The Declan Home Page

The Declan home page is up and running. I have also uploaded some prototype code to the SVN repository at RubyForge. This is just the beginning though. A release is a long way off.

Ari's Blog

My friend Ari Nordström mentioned Kallokain and me in a recent blog posting. I thought I'd return the favor. Ari's Blog has a much wider range of topics than Kallokain. Ari's recent posts span a range of topics, including XML (he is a very skilled information analyst and XML designer), Linux, authism, writing, chess, and how to counter pfishing and similar confidence scams.

Why not pop over and have a look? You might like it.

Spike Solutions

One of the things I like about Extreme Programming is the spike solution. A spike solution is a small, focused effort to explore a possibility, or solve a problem, by writing code. I worked a bit on Declan today. Declan is the declarative XML transformation system I wrote about in a previous post.

I have a very simple Declan prototype up and running, but it has one little snag: declarative Declan statements must be part of the body of a class declaration, or they won't work. I want to be able to write a file with rules, and then load those rules with a load or require statement. This would be easy to do, but for one thing: as the rules file is loaded, Ruby will execute the statements. This is not a problem per se, but the Declan statements will change the state of the application. I need to store that state, without having an object handy to store it in.

Being something of a Ruby tenderfoot, I'd like to explore the solution I have in mind before committing serious effort to refactoring the Declan prototype. A spike solution seems to be in order, but first well have a look at how the prototype works. You'll soon see why I am not happy with it.

In the prototype the Declan statements are declared in a superclass. The subclass, where the transformations are declared, creates a singleton instance of itself, and then call declan statements as methods on the singleton instance. This not only sounds complex, it looks horrible. The following is an excerpt from a test case for the prototype:

class TestTransformer < Test::Unit::TestCase
XPATH_ROOT = "root"
XPATH_CH_MATCHED = '//ch[@class="matched"]'

class Template < Declan::Transformer
attr_reader :rules, :node_hash

# Make a singleton
private_class_method :new
@@template = nil

def Template.create
@@template = new unless @@template
@@template
end

# Instantiate the singleton
Template.create

# These method calls are executed when the class is read in.

@@template.match(XPATH_ROOT) {|root|
<<-"ENDTEMPLATE"
<book>
#{@@template.apply_templates}
</book>
ENDTEMPLATE
}

@@template.match(XPATH_CH_MATCHED) {|ch|
<<-"ENDTEMPLATE"
<s status="#{ch.attributes['class']}">
#{@@template.apply_templates}
</s>
ENDTEMPLATE
}
end
#Setup and tests follow here
end

The first thing to do is to make a simplified model of the system I have now. This is the simplest thing I could come up with:

#! /usr/bin/ruby
class Flag
attr_reader :state

def initialize
@state = false
flip # Declarative statement
end

def flip
@state = !@state
end
end

flag = Flag.new()
puts flag.state

I have mashed things together a bit. The declarative statement (the flip method) is defined in the same class where it is used. I also dispensed with the declare a singleton nonsense by moving flip into the initialize method. Still, it's the same thing in principle, given that we have sufficiently flexible principles. BTW, when executed, the program prints true.

It would be nice if I could just break out the call to flip in a separate file, then include it, but I don't think that'l work. Nevertheless, I give it a try:


#! /usr/bin/ruby

class Flag
attr_reader :state

def initialize(rules)
@state = false
load rules
end

def flip
@state = !@state
end
end

flag = Flag.new('declarative_rules.rb')
puts flag.state

The declarative_rules.rb file looks like this:

flip


As expected, it does not work. The rules file executes as it loads, and at that time, there is of course no object to call the method on. However, the following does work:

#! /usr/bin/ruby

@state = false

def flip
@state = !@state
end

load 'declarative_rules.rb'

puts @state

The flip method is defined before I load the rules file, so the flip statement in the rules file can execute with no problem. The catch is that the @state property and the flip method are both defined in the top-level execution environment. This isn't good, because sometimes I do want to write Declan code inside an object. I just do not want to have to do it. However, I can move them out into a module of its own so I can mix it back in however I want. This is my mixin module:
@state = false

def flip
@state = !@state
end

And this is the final version of the program that executes the declarative code:

#! /usr/bin/ruby
require 'decltest4_mixin'
load 'declarative_rules.rb'
puts @state

Doesn't look like much, but it does tell me how to proceed when refactoring Declan. I'll move the definitions of the Declan statements from the Declan::Transformer class to a separate module that I can mix into any class that I want. I also know that I can dispense with the awkward singleton by moving the statements from the class body to the initialize method.

Now that I have done the spike, it seems very obvious. Can't understand why I didn't think of it right away...

Test::Unit::XML Quick Start Tutorial

If you are working with XML and Ruby, you might want to have a look at Test::Unit::XML, a unit test framework for XML documents and document fragments. The following is a short tutorial to help you get started.

This version of the tutorial is for Test::Unit::XML version 0.1.4.

Downloading and Installing

The easiest way to download and install Test::Unit::XML is to use gem, the Ruby package manager. All you need to do is this:

gem install testunitxml

You can also download Test::Unit::XML from the download page, and install manually.

If you install from the tarball or Zip archives, you need to:

1. Unpack the archive file
2. Run the script setup.rb from a console window.
   

Testing XML Documents

Let's try something easy, just to see how Test::Unit::XML works. Assume that you have two XML files that you wish to compare. Lets call them expected.xml and actual.xml. If you wish, you can pretend that actual.xml has been created by some software that you want to test.

Fire up a text editor and create a test case like this:

#! /usr/bin/ruby

@@lib_path = File.join(File.dirname(__FILE__), "..", "lib")
$:.unshift @@lib_path

require 'test/unit/xml'

class TestTestUnitXml < Test::Unit::TestCase
def setup
@expected = File.new("test/data/expected.xml")
@actual = File.new("test/data/actual.xml")
end

def test_io_objects
assert_xml_equal(@expected, @actual)
end
end

Note that when assert_xml_equal is given IO objects (File is a subclass of IO) they are converted to XML documents before the comparison is made.

In the current version of Test::Unit::XML, assert_xml_equal is the only available assertion. Luckily, assert_xml_equal is also the XML assertion you are likely to need the most. Also, assert_xml_equal can do a lot more than testing io objects. For example, you can test strings:

class TestTestUnitXml < Test::Unit::TestCase
def setup
@expected = %Q{<t:root xmlns:t="urn:x-hm:test" xmlns:x="urn:x-hm:test2" id="a" t:type="test1"/>}
@actual = %Q{<s:root xmlns:s="urn:x-hm:test" xmlns:x="urn:x-hm:test" id="a" s:type="test1"/>}
end

def test_strings
assert_xml_equal(@expected, @actual)
end
end

Note that the assertion succeds, even though the t and s namespace prefixes used by the root element are different. This is because the prefixes don't matter. Both root elements are bound to the same namespace, urn:x-hm:test, and that is what counts.

Also note that the xmlns:x declarations are different, yet the test succeeds. Why is that? It is because the x prefix is never actually used. Test::Unit::XML does not compare namespace declarations, only how the namespaces are used. There are reasons for this:

An XML processor may move declarations, and change prefixes, outside of programmer control. For example, XSLT processors may do this. Also, namespace declarations that are never actually used may suddenly appear as the result of a transformation. This does not affect how a document is processed in any way. Because of this, Test::Unit::XML completely ignores the namespace declarations. (If you need to be specific about which prefixes are used, on which element a namespace declaration ends up on, and whether there are any extraneous namespaces, you need to do DTD validation. Test::Unit::XML has no support for this, but you could try out the XML::Tools module at RubyForge.)

Testing REXML Nodes

The most flexible way to use the assert_xml_equal assertion is to feed it REXML nodes. You can compare any kind of REXML node, for example element nodes, CDATA nodes, comment nodes, etc. This is interesting, because it means you can compare pieces of documents. For example, given a document that contains a table with records, you could compare two records like this:

assert_xml_equal(table.elements["table/record[5]"], table.elements["table/record[7]"])

What About Doctype Declarations?

The current version of Test::Unit::XML does not validate documents against DTDs. However, the assert_xml_equal assertion does compare Doctype declarations, if they are present. (This is new from version 0.1.4.)

The online documentation describes precisely how doctype declarations are compared.

Transformations

A couple of months ago I wrote some XSLT stylesheets to extract information from W3 Schemas, and generate documentation from them in HTML format. I can't say I didn't have fun, because I did (I'm easy to amuse sometimes), but as always when I am working with XSLT, I wished I had a better way to do XML transformations.

XSLT is a powerful transformation language, but it is also very verbose, and the XML compliant syntax is definitely not designed for editing with a text editor. Also, you often need to write an application in some other language to get a complete application. Transformations are usually only a part of the job.

I had just become interested in Ruby, and also came into contact with Rake, a make-like (Ant-like, for Java programmers) build utility. Rake is written entirely in ruby, and has a very nice declarative syntax.

While I was immersed in Ruby/Rake, some of my XML neurons suddenly fired up. If Ruby is good for writing a declarative build language, maybe it is good for writing a declarative transformation language?

After a bit of fiddling, I had a working prototype of a very small DSL (Domain Specific Language) for making declarative XML transformations. I promptly named it Declan (Declarative Language), for want of better ideas. (I did have some worse ideas, but I won't discuss those.)

I also had a problem: writing an XML transformation language would obviously require a lot of unit testing. An excellent unit test framework, Test::Unit, is part of Ruby's standard library. There is no support for testing XML though. Nor was there any such framework available on the Web.

I briefly considered using XMLUnit, a Java test framework instead, but that would require me to mix Ruby and Java code in the same project. It felt cludgy, to say the least. Better to do things properly, and write a Ruby framework for testing XML. I realized that I wouldn't have to do it from the ground up. Test::Unit is well written, and quite easy to extend.

Said and done! Test::Unit::XML was born. Test::Unit::XML mixes in assertions for testing XML documents in the Test::Unit::TestCase class. I feel like a proud father today, because I have made the first code release on RubyForge.

The plan right now is to focus on Declan, and let the needs of Declan drive the development of Test::Unit::XML.

The funny thing is that as soon as I begun working on Test::Unit::XML, I realized that it is by far the most useful of the two projects. Every XML project needs an XML testing framework. The challenge now is to make Test::Unit::XML so good that Ruby programmers will want to use it.

Watch this blog for more about both the projects.

Getting Started

It's done! I have a blog.