What about complexity in software?

The September’s spotlight of Harvard Business Review (HBR.ORG) focused on managing complexity in business. There are a couple of interesting articles and I just can recommend reading them (unfortunately only excerpts are available for free online). The topic of complexity isn’t new to me. So, I found it an interesting exercise to discuss the topic of complexity in the context of software development and software products.

As preparation for the discussion we need two things. First of all we need to clarify our expectations with regard to predictability of software and the development process. Second, we need a brief understanding of what complexity means.

Let’s start with our expectations towards software products, as this is the easier task than the development process. What is our expectation towards software with regard to predictability? The answer is straight forward. In the vast majority of cases we want and expect software to work in a predictable way. The balance sheet of your company better is calculated in a predictable way. If you click on a button labeled “save” you expect that a file is saved to disk. You don’t want your computer to start an open ended discussion on whether he can give in on your request. We all know that our expectations are not always met. We will see in a minute why this is the case.

Now, let us turn to the software development process. This one is much trickier. Assume you already have a good understanding of what you want to develop – think of a PDF converter, a mail client, or an accounting system. At the end of the development process you expect to have the corresponding component. But is the process predictable? From a business perspective it would be favorable – a predictable process reduces the risk of failure. But development is not predictable. It becomes even more obvious if you start only from a rough idea. Imagine you want to develop a computer game but will decide based on market research and interviews what kind of game it will be. The outcome is not predictable. As a developer you do not expect this process to be predictable. From a classical business perspective this might be perceived unfavorable, and it is one of the main reasons behind mergers and acquisitions. If you acquire a product it is less risky than developing one (at least in the unreflected approach …).

Still, development is the source of innovation. Companies develop software because they assume the chance of making money exceeds the risk of failure. But as a serious company you need to contain the risk of failure. And here understanding complexity – or self-organization – is key.

From a mathematical point of view there are linear and non-linear systems. Linear systems are fully predictable. If you are at point A at time T1 you will be at point B at time T2. Mathematically not very spectacular. But from the software product perspective predictability is a highly aspired behavior. Complex systems on the contrary are a class of non-linear systems. Gökçe Sargut and Rita Gunther McGrath introduced the terms complicated and complex systems in their HBR article. This might be the attempt to make this topic more consumable for casual readers. Still, I think it is only of limited use. To embrace the concept of complexity – or self-organization – you need to understand it in more depth. Complicated systems in their view might be technically complicated, but still remain predictable. Think of a mechanical Swiss watch. It contains an amazing number of moving parts but still shows predictable behavior. If it is 10 o’clock now, you know the watch will read 11 o’clock in an hour (at least if it is precisely adjusted). Complex systems (at least in the interesting regime) are inherently unpredictable.

Complex systems are based on autonomously acting agents. For example cells, animals, humans, or artificial agents. The system is based on agent-agent interaction. The agents follow certain system-specific rules (e.g. if you get signal A, do action 1). Complex systems can operate in 3 regimes: the ordered, the chaotic, and the self-organized (or emergent) regime.

Order regime: In the ordered regime the non-linear effects are not present (e.g. by means of strong damping). The system basically behaves like a linear system and does not yield any interesting insight. Example of such a system would be a strongly hierarchical organization giving individuals no freedom at all. Or think of a marching army squad.

Chaotic regime: The opposite of the ordered regime is the chaotic one. Here, no order or patterns are left. The system is completely unpredictable. By all means, the chaotic regime is the least aspired one for any real system (maybe except for some scientists).

Self-organized/emergent regime: This regime is the most interesting one. The agents interact and follow their rules. They act autonomously and do not follow a central leader. This interaction can yield complex patterns but is inherently unpredictable. The textbook example per se is the ant colony. Ants do not follow any leader – this is why it is called a self-organized system. Each ant follows a set of rules. The interaction between ants is based on pheromone trails that guide the ants towards food, or else. In this way ant colonies yield a complex structure with dedicated areas for breeding, food storage, waste, etc. Other examples are herring or bird swarms. In short, the self-organized regime yields more than the sum of its parts. Sometimes people call it swarm intelligence. I think this is completely misleading and disregards the individual; it’s not what I would call intelligence.

This is complexity in very, very, very brief. If you are interested, I would recommend books like Hidden Order by John H. Holland, or Emergence by Steven Johnson. Still, my favorite one is Investigations by Stuart Kauffman. This one really inspired me.

Now, let us come back to the software context. Although we predictability is aspired for software products we are often annoyed by random behavior. What are the reasons for such unexpected behavior? The first thing to understand is that software – the code itself – behaves in a predictable way. This is the nature of algorithms. Anyone who has worked with software will find this puzzling. Software often does not seem to work predictably. There are two reasons (at least). On the one hand there are classical bugs in the code. These bugs might lead to unexpected results, still the behavior remains predictable. If you debug the code you can understand why the system shows the unexpected behavior. The same holds true for system-to-system communication; it is just a little more effort to find the bug. The second reason is more subtle. Software requires a hardware environment to run in. This environment can impose failure on the software. The hardware itself can break and make the software unable to operate. The other effect is that the computing resources required by the software exceed the physical limits of the hardware. Software can show random behavior for instance through out-of-memory errors. All such effects (even not an easy task) can be controlled by monitoring and testing efforts. It is the aim to operate a software system in the ordered regime. There is no room for complexity in the given sense here. And yes, I do not belief like Ray Kurzweil and others that a huge set of computers will start self-organizing across the internet to build a higher-level intelligence to take over humanity.

Now, let’s turn to the development process. We can rule out the chaotic regime to be aspired. But what about the ordered regime? From a competitive perspective we cannot seek the ordered regime. Although practically not possible, the ordered regime would yield only predictable solutions. Anyone with the same problem would come to the same (predictable) solution. There would be no room for a competitive advantage through better solutions. This does not mean that software development projects cannot be operated in a highly ordered or chaotic way. It just means that these regimes do not yield the best possible solutions (if any at all). We have to seek the self-organized regime to get better results in software development. What does this mean?

Software is developed by people. These people have different skills (development, architecture, product management, quality, etc.) and need to interact to find a solution to a given problem. The best solution is achieved by optimal utilization of the different skills. It requires creativity but also sticking to certain rules (e.g. staying within technological limits). But this is very much the self-organizing regime of a complex system. You don’t know what the final solution will look like – it’s not predictable. But the chance for a good or great solution is much higher in this regime. This makes the success of some start-up companies – smart people amplifying their impact through interaction in a small and agile team. A method trying to industrialize this innovation process is called Design Thinking. This approach emphasizes the need for diversity and feedback/interaction within the team.

Such a start-up mode is often aspired by software companies and projects. The issue arises with the size of the team and the expectation with regard to the output. To limit the risk of failure (which is inherent to the start-up mode) we need mechanism – rules – that are compatible with the self-organizing regime. Wrong rules easily drive software development projects outside this regime. But this will be the topic of future posts. Just one example. Everybody who has worked with off-shore development or projects spread across different location made the experience that this is much less efficient and effective than having all developers in a single location. From a complex system’s perspective this is clear. The interaction between people (agents) – an essential ingredient of complex systems – is hampered by the distance. Self-organization and amplification of ideas can not emerge.

Hope this makes the relevance of complex system theory for software development a little more transparent and tangible.