Why software engineering is different
Project based “Design first and build it once” approaches are a waste when creating software. Software engineering benefits hugely from an agile approach to product development - not just because software lets us tackle more complex problems, but also because of the different constraints that software development faces compared to other engineering disciplines. Software is wonderfully volatile and easy to change compared to other engineering disciplines, so a different way of building products is necessary.
Since software is comparatively easy to adapt, projects often start with a much fuzzier understanding of the underlying challenge than projects in civil or mechanical engineering. We often try to solve complex problems with software, and this means that the outcome of design decisions cannot be predicted reliably until they are tried out. Consequently, we software engineers often have to figure out the exact requirements as we go along. Often, these are not just minor questions but fundamental ways on how the software should work. Traditional “design first and build once” approaches are very bad at coping with the “we figure it out as we go along” attitude needed to tackle such kinds of problems. This is why agile software development was born 20 years ago. Instead of elaborate plans, agile development relies on fast iterations and tight feedback loops to sharpen requirements while developing a product incrementally.
It’s not just the complexity of the underlying problems, that make an agile approach a prime choice when developing software. The internal mechanics of software development itself fit very well to an agile approach. The factors and constraints that make classical design-first approaches to project management a necessary - and thus good - choice in many fields of engineering are just not given for software development.
Building software is ridiculously cheap
Disciplines like civil engineering rely on a plan-design-build approach because in addition to the costs of planning and design, actually building a bridge or house is expensive. Many people spend a significant time working on a construction site, and the vast amount of of materials present on a construction site are just some of the obvious cost factors. Count in the running costs of all the machines and tools that are used by the builders and an hour of work can get quite expensive. It would just would not do to build the bridge twice, just because of a minor design change. While new technologies such as prefabs have an impact on the costs, building and assembling everything, remains major point in the budget.
As a contrast, building software is ridiculously cheap. Buy a notebook, install a compiler, press a button even huge programs are magically assembled. Even whole operating systems can nowadays be built with a single command. The hard part of creating software is designing it and putting this design in an unambiguous description which the machine can understand - also known as programming or coding. Compared to other engineering disciplines, this is a very radical shift in the internal distribution of cost and effort in a project.
Building multiple revisions of physical objects is not efficient. Once a machine is built, it occupies storage space, which might incurr additional rental costs. So the idea is to push the done product to the customer as fast as possible and have almost no stock - also known as just-in-time-production. A worst case scenario is having many stocked machines that have to be retrofitted or scrapped without having being used once because of a design change. On the other hand storing as many software revisions as I want is ultra cheap. An external disk of a terabyte of space costs roughly a hundred bucks and can literally store tens of thousands of revisions of a software1. Consequently producing many versions of a programs with only slight changes in between is possible at no additional storage or building cost - also known as iterative development. There is of course the design cost of all these variations, but the design is needed anyway to tackle any new problem - whether you are doing software or not.
In modern software development we are not just storing the product. We can store the fully-automated workshop along with the product so we can ensure that the product can be built and run in the future as well. What I’m referring to is of course containerization which made replicating software build ridiculously easy and which is a huge game changer since it gained popularity roughly 10 years ago. Traditional engineering disciplines do not have that possibility, construction sites are dismantled when the project is done and once a workshop is set up for a certain kind of production it usually stays that way for some time.
All these factors contribute to the fact that software products - or products that require a larger part of software to run - have to be developed and built differently than traditional products to reap all the benefits. Software is inherently able - and often needed - to adapt to a constantly changing environment, not just in the early phase but over the whole lifetime of a product. Just think of all the necessary security patches one might have to deliver during the lifetime of a software product. Then there is software development itself that evolves quite fast. We went from procedural coding over object oriented to functional programming in a few decades. Each step fundamentally changing the way software is designed. A new generation of coders that maintain a product might have a completely different design approach for new features.
Doing a plan first and build-it-once-and-your-done approach to building software products is like driving a car with the handbrake on. It just does not leverage the huge power that lies in the inherent flexibility of software development. It is actually even worse: Not accounting for the fast evolution of the whole software universe results in products that are of inferior quality and expensive to maintain over time.
Even in 2021 many try software development with a front-loading approach
Twenty years ago the Agile manifesto was published, with the aim of unlocking exactly this potential of software engineering. Since then “agile” has become THE buzzword for lots of companies, but apart from pure software companies, its application is rare. Even in 2021 many companies approaches software development with traditional, front-loading project management methods. Almost all customer negotiations I had in the last few years started from the position of a fixed-time, fixed-scope and that at the start of the project “the requirements and parts of the design are largely done”. The battles to move away from that position towards a more iterative and incremental approach are often hard-fought, but necessary to at least partly benefit from the volatility of software development.
Software has the in-built potential for evolutionary growth based on customer feedback, yet lots of companies still think in terms of “feature freezes” and “development-phase” and “maintenance-phase”. If you want to get the most out of software, abandoning this approach inf favor of a “living product” that grows more organically is the way to go. Only then will software development truly become a way to earn money instead of a necessary but expensive nuisance in order to make the product functioning. Just adding software development to an existing engineering and project management process will end in expensive frustration. By understanding that software development is a unique discipline that works from a completely different internal cost structure a lot can be gained. So take a step back and look at your software projects, are they geared towards high mutability and cheap construction or is it rather the other way round?
-
Professionally run and backed up solutions for storing data are slightly more expensive, but still quite cheap. ↩