Collapse of Complexity
In this post Clay Shirky discusses the collapse of complex business models and Tainter’s book The Collapse of Complex Societies.
…a group of people, through a combination of social organization and environmental luck, finds itself with a surplus of resources. Managing this surplus makes society more complex—agriculture rewards mathematical skill, granaries require new forms of construction, and so on.
Early on, the marginal value of this complexity is positive—each additional bit of complexity more than pays for itself in improved output—but over time, the law of diminishing returns reduces the marginal value, until it disappears completely. At this point, any additional complexity is pure cost.
….
When the value of complexity turns negative, a society plagued by an inability to react remains as complex as ever, right up to the moment where it becomes suddenly and dramatically simpler, which is to say right up to the moment of collapse. Collapse is simply the last remaining method of simplification.
As Shirky points out in his post, the same thing can happen in business. I believe something similar can happen with any over complex structure such as some software projects. The FAA famously killed their modernization project in the late 1980’s due to this. Way back in 1975 Fred Brooks in his book The Mythical Man-Month observed, “Adding people to a late project only makes it later.” In reference to theoretical physics Einstein said, “Everything should be made as simple as possible, but not simpler.” Sounds like good advice in general.
The study of failed software projects and systems should be part of our standard training for a software engineer. In aviation accident analysis is an important area of study and civil engineers also do this. I’m not the only pilot who’s read years of NTSB reports. However in software we have nothing similar. Fred Brooks’s book is a good start; it’s a classic. But due to corporate embarassement and proprietary property issues, there’s no similar study in software engineering.
This lack forces us individually to replicate all of our predecessors mistakes. The progress of our field suffers.