Continuous integration, delivery and deployment
Agile development is fundamentally about getting customer feedback quickly and frequently. In my experience, this leads to better software: it minimises the cost of course corrections, reduces useless or unnecessary features, and, ultimately, makes for happier customers and developers. This approach is summarised in the original principles of agile development:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software
- Deliver working software frequently… with a preference to [a] shorter timescale
- Working software is the primary measure of progress
Back in the early 2000s, when the Agile Manifesto was written, delivering software “from a couple of weeks to a couple of months” was bold and aspirational. In today’s world of web development, we can (and should) do much better.
Case study
In 2020, the UK government implemented a programme of changes to the deployment processes of the GOV.UK portal, a suite of 50 web apps covering a range of functions. Before these changes, each deployment was a manual, time-consuming process, and deployments were often done in batches. There was an average delay of three days between making a change and deploying it. If any of these large batches introduced a fault, it was difficult to isolate the specific change that caused it. The delay meant that a new developer would often be tasked with diagnosing and fixing the issue, without the necessary context that the original developer had.
To address these issues, the team introduced a completely new approach to deployment. First, a developer making a change would ask for a code review, and automated tests would be run against the branch. Then, if approved, the change would then be deployed to the http://GOV.UK integration environment, where further tests would probe the running app and those it communicated with. If these tests passed, the change would automatically be deployed to the http://GOV.UK staging environment, where the process repeated before deploying to the live site.
The new approach was trialled on a small subset of services. There were no production incidents introduced by the trials, so continuous delivery was rolled out across the portal. The average time from change to deployment is now half a day, with much less manual intervention than before.
The change was a success, according to the team:
[The new process] has ensured that our apps are fully patched and up to date. It used to take far too long to manually deploy changes, which meant that they would be delayed for days or even weeks.
A brief history of deployments
Of course, the ability to perform frequent, lightweight deployments like this is relatively new. For most of the history of computing, software deployment has been manual and time-consuming.
The Colossus, the codebreaking system built in Britain during the second world war, is widely regarded as the world’s first digital electronic computer. A team of mathematicians, scientists and engineers including Alan Turing ‘programmmed’ this great machine using physical plugs and switches: there was no concept of a stored program. This concept was introduced in the Manchester Mark I (nicknamed Baby), developed at Manchester University in Britain in 1948. Software was stored on paper tape as a series of punched holes, similar in concept to the punched cards used by Charles Babbage’s Analytical Engine, and stored in a memory implemented as specialised cathode ray tubes called Williams-Kilburn tubes.
Still, there was no concept of ‘installation’: programs had to be loaded into memory from the paper tape before use. By the 1960s, mainframes like the IBM System/360 were commercially available, with magnetic hard drives to install software onto. Installation was typically carried out manually from physical media like punch cards and magnetic tapes, but later System/360s supported remote distribution and installation of patches.
By the late 1970s, computers were starting to enter the home. Early consoles like the Atari 2600 (1977) had no hard drive: games were loaded from cartridges. Home computers like the ZX80 (1980) loaded programs from magnetic cassette tapes, or users typed in the BASIC code from books or magazines. Personal computers like the IBM PC XT (1983) were starting to feature more and more in the office, with small internal hard drives and software installed from floppy disks.
In 1989, the World Wide Web was born, paving the way for home and office users to download software. The next decade also saw the birth of online services like Yahoo (1994), Amazon (1995) and Hotmail (1996). These services were installed on physical servers and updated manually via FTP or similar protocols; updates typically required server restarts and downtime.
In 2006, Amazon Web Services was born, and thus began the era of cloud computing. This enabled companies to deploy web services much more efficiently, spinning up multiple server instances to enable load-balancing and proper zero-downtime deployments. In 2009, John Allspaw and Paul Hammond gave a talk called 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr, and by the early 2010s, Facebook was doing this on a huge scale.
Current state
Performing multiple deployments per day has been possible for well over a decade; however, even today, teams that do this remain in a relatively small minority. According to the Continuous Delivery Foundation’s State of CI/CD Report 2024, in Q1 2024, only 9% of surveyed teams made multiple deployments per day, whilst 40% deployed less frequently than once a month. 11% of teams had an average service recovery time of under one hour, whilst 41% averaged over a week. These are sobering statistics.
That being said, it’s important to remember that not all teams are in a position to deliver software in such an agile manner. For example, device drivers often need to be backwards-compatible with numerous old devices, and the associated testing requirements make release cycles necessarily long. Not all web services can be SaaS: customers may have regulatory requirements regarding where their data is stored, or simply wish to control the upgrade cycle. A service provider can end up juggling dozens of production versions for different customers. Even so, there are things every team can do to improve matters.
The better deployments toolbox
The first tool in our toolbox is continuous integration: developers regularly merge their code changes into a central repository, triggering builds and automated tests. This provides a minimal level of code hygiene, as it ensures the trunk branch builds and basic use-cases work as expected. All teams can do this, regardless of the constraints their business or customers put on them.
The next tool is continuous delivery: after CI checks are complete, the code changes are automatically deployed to a development or staging environment, where they can be validated before being pushed to production. This requires the team have control of the deployment cycle, and works best with Software as a Service (SaaS). With this approach, all environments run the same version of the code (eventually), differing only in data and configuration. This makes debugging production bugs much easier, without the need to juggle version tags. Daily releases become possible so long as the QA engineers and product owner collaborate efficiently with the team. This is the approach taken by the UK government (as described earlier in the post).
Finally, the most mature teams implement continuous deployment. The difference between continuous delivery and continuous deployment is that continuous delivery has a big red button to update to production. With continuous deployment, the big red button is gone, and production happens automatically. This requires more discipline than continuous delivery, as all pushes to the trunk must be production-ready. Code-reviews, automated tests and over-the-shoulder reviewing/testing all help to make this possible. Teams that practice continuous deployment can easily perform multiple deployments per day.
In my experience, adoption of these practices is a step-by-step process. As teams learn to do continuous integration, they develop the skills necessary to adopt continuous delivery. As they master continuous delivery, this prepares them for continuous deployment.
Continuous delivery and deployment in detail
The first artefact of continuous delivery is the feature branch. This branch is either not deployed at all, or deployed to a transient environment by the developer. It is inherently unstable. Pull requests are created from feature branches, and continuous integration pipelines run automated tests and other checks.
Pull requests are merged to the trunk branch (typically called main in git). On merge, the trunk is then deployed to a development environment. This environment is more stable than a feature branch, but less stable than production. QA may perform exploratory testing, and product owners may review new features.
A one-click deploy process promotes this version to production, which should remain strong and stable. (In continuous deployment, the middle step is omitted: the code is deployed simultaneously to the dev and prod environments on each push to the trunk.)
Baby steps
The UK government did not implement full continuous deployment, but they did make significant improvements to the service they provide to their citizens. Whilst only a minority of teams enjoy the required circumstances (and maturity) to implement full continuous deployment, all teams will benefit from implementing as many of these improvements as their customers and domain allow. Any team can benefit from continuous integration. Any SaaS team should aspire to implementing continuous deployment, and many teams building custom on-prem solutions could integrate elements of continuous delivery into their pipelines. Do not let perfect be the enemy of good (or better)!
Further reading
Despite being two decades old, the Extreme Programming site is still a great place for tips on how to be a better developer.
