Coffee, LEGO movies, questionnaires, and complexity theory


I recently had a cup of coffee with a friend, and the discussion turned to the difference between complicated and complex, and why the difference is important.

I have had reason to think about that recently, so I had a couple of examples fresh in my mind, both relating to questionnaires and surveys.

As it turns out, many questionnaires you are asked to fill out have a common design mistake: The assumption that the subject under investigation is complicated, rather than complex. It is an easy mistake to make. The result is increased risk that the survey points you in the wrong direction.

Let's briefly define what we are talking about before digging in to the meat of the matter:
  • Complicated systems have many parts, but they also have predictable cause and effect relationships. For example, a mechanical watch is complicated. It is also predictable. It runs like...well, it runs like clockwork.
  • Complex systems have parts that can adapt to the behavior of other parts in the system. A family is a complex system. All family members both react and adapt to what other family members do. Business organisations, countries, teams and workgroups, ecological systems, the scouts, my photo club, and aquariums are also complex systems.

You can pick a mechanical watch apart and study each piece, that is analyse it, to figure out how it works.

On the other hand, studying each member of a family, or a software development team, or each fish in an aquarium, will not necessarily tell you how the system as a whole will work. A family has emergent properties, properties that belong to the family as a whole, but not to any of its members.

LEGO movies and unpredictability

Here is an emergent property of the system consisting of my eight year old son and me: Stop motion movie making.


My son asked me if we could make a LEGO movie. I said yes, of course, and we created the short movie above. It would be impossible to predict in advance that my son and I would produce a short LEGO movie featuring Thor and The Hulk.

In retrospect, it does not seem farfetched at all that we would do such a thing. It would be easy to construct a Future Reality Tree explaining why and how we did it. However, the tree would have to be created afterwards. It would be impossible to construct a tree that accurately predicts what we will do.

What on Earth does this have to do with surveys and questionnaires? As it turns out, a lot!

Questionnaires: The art of asking the wrong questions

To find out something about a complicated system, you can ask a question about a part of the system. If you want to know more, you can continue to ask questions about parts of the system. Eventually, you can compile the answers, and they will tell you a lot about the system as a whole.

With a complex system, that does not work very well. Knowing each part won't tell you the emergent properties of the system. Another problem is that with a complex system, you do not necessarily know which parts and properties of those parts, that are important to the functioning of the whole.

Systems where humans interact, are complex systems, but questionnaires are very often designed with the implicit assumption they are complicated, or even simple. Thus, most questionnaires, even the ones you pay specialists to create, are designed wrong. They do not tell you what you need to know!
As the illustration above shows, asking many specific questions means you get specific knowledge of the things you assume are important. however, you have no real basis for making these assumptions, because you haven't studied the system yet.

For example, some time ago a coffee shop I sometimes visit made a survey using touch screen computers and a set of specific questions. The questions were about the quality of service at the counter, whether the personnel behavied in an appropriate manner, whether there were cups and plates left on unoccupied tables in sight of the computer, and other things the management wanted to know.

As a fairly frequent guest, I noted that all of the things the questionnaire had questions about worked very well. There were problems, but the questionnaire did not mention them.

For example, there were several electrical outlets that were damaged well beyond the point of being dangerous, the toilets often ran out of soap and toilet paper, the free WiFi-system did not work. While  there was no problem with cups and plates lying around near the computer terminal, there were often several tables on the second floor that could not be used because they were covered in cups, plates, and glasses.

Nobody asked about it, because they did not know they should. The things they did ask about were the things they already had control of.

Thus, the questionnaire was all but useless as a tool for improving the café. All it could do was confirm that things the management had focused on in the past were ok.

The questionnaire created a false sense of having everything under control, which reduced the incitament to do real improvement.

Query Fatigue

Thinking requires a lot of energy, and the human brain has very limited energy reserves. This means a questionnaire with many questions will tire the brains answering the questions. Thus, the quality of the replies will degrade significantly, so that replies to question 26 will be much less trustworthy than replies to question 3.

Many people will of course opt out of replying to a questionnaire altogether, if it has too many questions.

To know more, you must ask less!

So, if a questionnaire gets worse the more comprehensive it is, what can you do about it?

Well, if asking more questions makes the questionnaire worse, then you can make it better by asking fewer questions.

If you ask only a few questions, then obviously you must ask the right ones, or you will learn little of significance. You cannot know what is important to ask, but there are people who know: The people answering the questionnaire. They probably do not know it as individuals, but collectively they do.

How can you tap into that knowledge? You can ask broad questions.
If you ask broad, open-ended questions like:

  • What is the single most important thing about X that we should improve?
  • What are the most important problems with X?

Then you will get the respondents to tell you what they believe is important, rather than telling you their beliefs about what you believe is important to them.

The difference is quite important.

There are many ways of doing this. Personally, I like the Crawford Slip brainstorming method. I also use a modified form of Net Promoter Score. (I had to modify it, because the original version of NPS botches the statistics and makes the assumption all systems are complicated.)

Scope and analysis paralysis

There are two important questionnaire design problems I am saving for another day and another post:

  • Scope: Which people should you ask? This is sometimes obvious. On the other hand, the obvious answer is often wrong, so you need to give this some thought.
  • Analysis paralysis: What do you do once you get the responses to your questionnaire? How do you know which answers are important, and which ones are not?
With those two questions I bid you farewell, for now. Until next time, think about questionnaires and surveys in your own organisation. Did they really tell you what you need to know? Were the big problems solved? If not, what can you do about it?

Comments

Post a Comment

Popular posts from this blog

Waterfall vs. Agile: Battle of the Dunces or A Race to the Bottom?

Image
The major software methodology wars since the mid 1950’s. Someone was wrong on the Internet, so here we go… In a recent HBR article, It’s Time to End the Battle Between Waterfall and Agile , the author sets up a false premise: There is a war between Waterfall methodology and Agile. The war must end. And finally, you can combine the approaches to get the best of both worlds. This sounds good, but the article is based on a misunderstanding of both Waterfall and Agile. Also, there is no war between Waterfall methodology and Agile. There can’t be, because Waterfall methodology does not exist! Waterfall is a name for large projects that failed in the 1960’s. Waterfall was never a methodology, but a failure to apply the methodologies that existed back then. As I will show towards the end of this article, at least one of the “successful” Waterfall projects mentioned in the HBR article was neither successful, nor a Waterfall project. I got invaluable help from Alistair Cockburn. He fact che
Read more

Performance Evaluations, Business Strategy, and Agile Methodologies

Image
It is performance evaluation time in many companies. This can be stressful, both to the people being evaluated, and the people doing the evaluation. In companies adopting agile software development methods, the tension can be extraordinary. Individual performance evaluations run counter to agile philosophy, which emphasizes team performance over individual performance. However, managers and corporate leaders need to take a few steps back, and consider the impact performance evaluations have on the organization as a whole. Especially now, in the midst of a recession, it is important to look at a companies current policies to see if they can be improved, or if they are actually holding the company back. So, how can a manager evaluate policy? Performance evaluation policies can serve as an excellent example. I'll confine myself to discussing the so-called ‘rank and yank’ methods. These are performance reviews were employees are ranked using a forced ranking system. It usually looks so
Read more

Agile Requirements Structures, Part 1

Image
If you have worked in large Agile projects or programs, you may have noticed that there is often quite a bit of confusion about requirements, what the requirement types are, their purpose, how to write them, how to use them, and above all, what not to do with them. There are many causes for this confusion. Here are some of the more common I have seen: Nobody in the organization has read up on the requirements model it is using, so everyone makes their own interpretation. The organization deviates from a standard requirements model, but nobody knows which standard model the organization is deviating from, and/or there is no agreement about how they deviate from it. The organization has not documented its own requirements model very well. The organization mixes two or more different requirements models, often without realizing it. The organization has documented its requirements model, but finding the documentation is an epic project in itself, on par with when Henry Sta
Read more