Continuous Integration

Put everything in a version controlled mainline

These days almost every software team keeps their source code in a version control system, so that every developer can easily find not just the current state of the product, but all the changes that have been made to the product. Version control tools allow a system to be rolled back to any point in its development, which can be very helpful to understand the history of the system, using Diff Debugging to find bugs. As I write this, the dominant version control system is git.

But while version control is commonplace, some teams fail to take full advantage of version control. My test for full version control is that I should be able to walk up with a very minimally configured environment - say a laptop with no more than the vanilla operating system installed - and be able to easily build, and run the product after cloning the repository. This means the repository should reliably return product source code, tests, database schema, test data, configuration files, IDE configurations, install scripts, third-party libraries, and any tools required to build the software.

You might notice I said that the repository should return all of these elements, which isn't the same as storing them. We don't have to store the compiler in the repository, but we need to be able to get at the right compiler. If I check out last year's product sources, I may need to be able to build them with the compiler I was using last year, not the version I'm using now. The repository can do this by storing a link to immutable asset storage - immutable in the sense that once an asset is stored with an id, I'll always get exactly that asset back again. I can also do this with library code, providing I both trust the asset storage and always reference a particular version, never “the latest version”.

Similar asset storage schemes can be used for anything too large, such as videos. Cloning a repository often means grabbing everything, even if it's not needed. By using references to an asset store, the build scripts can choose to download only what's needed for a particular build.

In general we should store in source control everything we need to build anything, but nothing that we actually build. Some people do keep the build products in source control, but I consider that to be a smell - an indication of a deeper problem, usually an inability to reliably recreate builds. It can be useful to cache build products, but they should always be treated as disposable, and it's usually good to then ensure they are removed promptly so that people don't rely on them when they shouldn't.

A second element of this principle is that it should be easy to find the code for a given piece of work. Part of this is clear names and URL schemes, both within the repository and within the broader enterprise. It also means not having to spend time figuring out which branch within the version control system to use. Continuous Integration relies on having a clear mainline - a single, shared, branch that acts as the current state of the product. This is the next version that will be deployed to production.

Teams that use git mostly use the name “main” for the mainline branch, but we also sometimes see “trunk” or the old default of “master”. The mainline is that branch on the central repository, so to add a commit to a mainline called main I need to first commit to my local copy of main and then push that commit to the central server. The tracking branch (called something like origin/main) is a copy of the mainline on my local machine. However it may be out of date, since in a Continuous Integration environment there are many commits pushed into mainline every day.

As much as possible, we should use text files to define the product and its environment. I say this because, although version-control systems can store and track non-text files, they don't usually provide any facility to easily see the difference between versions. This makes it much harder to understand what change was made. It's possible that in the future we'll see more storage formats having the facility to create meaningful diffs, but at the moment clear diffs are almost entirely reserved for text formats. Even there we need to use text formats that will produce comprehensible diffs.

Automate the Build

Turning the source code into a running system can often be a complicated process involving compilation, moving files around, loading schemas into databases, and so on. However like most tasks in this part of software development it can be automated - and as a result should be automated. Asking people to type in strange commands or clicking through dialog boxes is a waste of time and a breeding ground for mistakes.

Computers are designed to perform simple, repetitive tasks. As soon as you have humans doing repetitive tasks on behalf of computers, all the computers get together late at night and laugh at you.

-- Neal Ford

Most modern programming environments include tooling for automating builds, and such tools have been around for a long time. I first encountered them with make, one of the earliest Unix tools.

Any instructions for the build need to be stored in the repository, in practice this means that we must use text representations. That way we can easily inspect them to see how they work, and crucially, see diffs when they change. Thus teams using Continuous Integration avoid tools that require clicking around in UIs to perform a build or to configure an environment.

It's possible to use a regular programming language to automate builds, indeed simple builds are often captured as shell scripts. But as builds get more complicated it's better to use a tool that's designed with build automation in mind. Partly this is because such tools will have built-in functions for common build tasks. But the main reason is that build tools work best with a particular way to organize their logic - an alternative computational model that I refer to as a Dependency Network. A dependency network organizes its logic into tasks which are structured as a graph of dependencies.

A trivially simple dependency network might say that the “test” task is dependent upon the “compile” task. If I invoke the test task, it will look to see if the compile task needs to be run and if so invoke it first. Should the compile task itself have dependencies, the network will look to see if it needs to invoke them first, and so on backwards along the dependency chain. A dependency network like this is useful for build scripts because often tasks take a long time, which is wasted if they aren't needed. If nobody has changed any source files since I last ran the tests, then I can save doing a potentially long compilation.

To tell if a task needs to be run, the most common and straightforward way is to look at the modification times of files. If any of the input files to the compilation have been modified later than the output, then we know the compilation needs to be executed if that task is invoked.

A common mistake is not to include everything in the automated build. The build should include getting the database schema out of the repository and firing it up in the execution environment. I'll elaborate my earlier rule of thumb: anyone should be able to bring in a clean machine, check the sources out of the repository, issue a single command, and have a running system on their own environment.

While a simple program may only need a line or two of script file to build, complex systems often have a large graph of dependencies, finely tuned to minimize the amount of time required to build things. This website, for example, has over a thousand web pages. My build system knows that should I alter the source for this page, I only have to build this one page. But should I alter a core file in the publication tool chain, then it needs to rebuild them all. Either way, I invoke the same command in my editor, and the build system figures out how much to do.

Depending on what we need, we may need different kinds of things to be built. We can build a system with or without test code, or with different sets of tests. Some components can be built stand-alone. A build script should allow us to build alternative targets for different cases.

Make the Build Self-Testing

Traditionally a build meant compiling, linking, and all the additional stuff required to get a program to execute. A program may run, but that doesn't mean it does the right thing. Modern statically typed languages can catch many bugs, but far more slip through that net. This is a critical issue if we want to integrate as frequently as Continuous Integration demands. If bugs make their way into the product, then we are faced with the daunting task of performing bug fixes on a rapidly-changing code base. Manual testing is too slow to cope with the frequency of change.

Faced with this, we need to ensure that bugs don't get into the product in the first place. The main technique to do this is a comprehensive test suite, one that is run before each integration to flush out as many bugs as possible. Testing isn't perfect, of course, but it can catch a lot of bugs - enough to be useful. Early computers I used did a visible memory self-test when they were booting up, which led me referring to this as Self Testing Code.

Writing self-testing code affects a programmer's workflow. Any programming task combines both modifying the functionality of the program, and also augmenting the test suite to verify this changed behavior. A programmer's job isn't done merely when the new feature is working, but also when they have automated tests to prove it.

Over the two decades since the first version of this article, I've seen programming environments increasingly embrace the need to provide the tools for programmers to build such test suites. The biggest push for this was JUnit, originally written by Kent Beck and Erich Gamma, which had a marked impact on the Java community in the late 1990s. This inspired similar testing frameworks for other languages, often referred to as Xunit frameworks. These stressed a light-weight, programmer-friendly mechanics that allowed a programmer to easily build tests in concert with the product code. Often these tools have some kind of graphical progress bar that is green if the tests pass, but turns red should any fail - leading to phrases like “green build”, or “red-bar”.

The test of such a test suite is that we should be confident that if the tests are green, then no significant bugs are in the product. I like to imagine a mischievous imp that is able to make simple modifications to the product code, such as commenting out lines, or reversing conditionals, but is not able to change the tests. A sound test suite would never allow the imp to do any damage without a test turning red. And any test failing is enough to fail the build, 99.9% green is still red.

Self-testing code is so important to Continuous Integration that it is a necessary prerequisite. Often the biggest barrier to implementing Continuous Integration is insufficient skill at testing.

That self-testing code and Continuous Integration are so tied together is no surprise. Continuous Integration was originally developed as part of Extreme Programming and testing has always been a core practice of Extreme Programming. This testing is often done in the form of Test Driven Development (TDD), a practice that instructs us to never write new code unless it fixes a test that we've written just before. TDD isn't essential for Continuous Integration, as tests can be written after production code as long as they are done before integration. But I do find that, most of the time, TDD is the best way to write self-testing code.

The tests act as an automated check of the health of the code base, and while tests are the key element of such an automated verification of the code, many programming environments provide additional verification tools. Linters can detect poor programming practices, and ensure code follows a team's preferred formatting style, vulnerability scanners can find security weaknesses. Teams should evaluate these tools to include them in the verification process.

Of course we can't count on tests to find everything. As it's often been said: tests don't prove the absence of bugs. However perfection isn't the only point at which we get payback for a self-testing build. Imperfect tests, run frequently, are much better than perfect tests that are never written at all.

Everyone Pushes Commits To the Mainline Every Day

No code sits unintegrated for more than a couple of hours.

-- Kent Beck

Integration is primarily about communication. Integration allows developers to tell other developers about the changes they have made. Frequent communication allows people to know quickly as changes develop.

The one prerequisite for a developer committing to the mainline is that they can correctly build their code. This, of course, includes passing the build tests. As with any commit cycle the developer first updates their working copy to match the mainline, resolves any conflicts with the mainline, then builds on their local machine. If the build passes, then they are free to push to the mainline.

If everyone pushes to the mainline frequently, developers quickly find out if there's a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it's easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve.

Conflicts in the codebase come in different forms. The easiest to find and resolve are textual conflicts, often called “merge conflicts”, when two developers edit the same fragment of code in different ways. Version-control tools detect these easily once the second developer pulls the updated mainline into their working copy. The harder problem are Semantic Conflicts. If my colleague changes the name of a function and I call that function in my newly added code, the version-control system can't help us. In a statically typed language we get a compilation failure, which is pretty easy to detect, but in a dynamic language we get no such help. And even statically-typed compilation doesn't help us when a colleague makes a change to the body of a function that I call, making a subtle change to what it does. This is why it's so important to have self-testing code.

A test failure alerts that there's a conflict between changes, but we still have to figure out what the conflict is and how to resolve it. Since there's only a few hours of changes between commits, there's only so many places where the problem could be hiding. Furthermore since not much has changed we can use Diff Debugging to help us find the bug.

My general rule of thumb is that every developer should commit to the mainline every day. In practice, those experienced with Continuous Integration integrate more frequently than that. The more frequently we integrate, the less places we have to look for conflict errors, and the more rapidly we fix conflicts.

Frequent commits encourage developers to break down their work into small chunks of a few hours each. This helps track progress and provides a sense of progress. Often people initially feel they can't do something meaningful in just a few hours, but we've found that mentoring and practice helps us learn.

Every Push to Mainline Should Trigger a Build

If everyone on the team integrates at least daily, this ought to mean that the mainline stays in a healthy state. In practice, however, things still do go wrong. This may be due to lapses in discipline, neglecting to update and build before a push, there may also be environmental differences between developer workspaces.

We thus need to ensure that every commit is verified in a reference environment. The usual way to do this is with a Continuous Integration Service (CI Service) that monitors the mainline. (Examples of CI Services are tools like Jenkins, GitHub Actions, Circle CI etc.) Every time the mainline receives a commit, the CI service checks out the head of the mainline into an integration environment and performs a full build. Only once this integration build is green can the developer consider the integration to be complete. By ensuring we have a build with every push, should we get a failure, we know that the fault lies in that latest push, narrowing down where have to look to fix it.

I want to stress here that when we use a CI Service, we only use it on the mainline, which is the main branch on the reference instance of the version control system. It's common to use a CI service to monitor and build from multiple branches, but the whole point of integration is to have all commits coexisting on a single branch. While it may be useful to use CI service to do an automated build for different branches, that's not the same as Continuous Integration, and teams using Continuous Integration will only need the CI service to monitor a single branch of the product.

While almost all teams use CI Services these days, it is perfectly possible to do Continuous Integration without one. Team members can manually check out the head on the mainline onto an integration machine and perform a build to verify the integration. But there's little point in a manual process when automation is so freely available.

(This is an appropriate point to mention that my colleagues at Thoughtworks, have contributed a lot of open-source tooling for Continuous Integration, in particular Cruise Control - the first CI Service.)

Fix Broken Builds Immediately

Continuous Integration can only work if the mainline is kept in a healthy state. Should the integration build fail, then it needs to be fixed right away. As Kent Beck puts it: “nobody has a higher priority task than fixing the build”. This doesn't mean that everyone on the team has to stop what they are doing in order to fix the build, usually it only needs a couple of people to get things working again. It does mean a conscious prioritization of a build fix as an urgent, high priority task

Usually the best way to fix the build is to revert the latest commit from the mainline, taking the system back to the last-known good build. If the cause of the problem is immediately obvious then it can be fixed directly with a new commit, but otherwise reverting the mainline allows some folks to figure out the problem in a separate development environment, allowing the rest of the team to continue to work with the mainline.

Some teams prefer to remove all risk of breaking the mainline by using a Pending Head (also called Pre-tested, Delayed, or Gated Commit.) To do this the CI service needs to set things up so that commits pushed to the mainline for integration do not immediately go onto the mainline. Instead they are placed on another branch until the build completes and only migrated to the mainline after a green build. While this technique avoids any danger to mainline breaking, an effective team should rarely see a red mainline, and on the few times it happens its very visibility encourages folks to learn how to avoid it.

Keep the Build Fast

The whole point of Continuous Integration is to provide rapid feedback. Nothing sucks the blood of Continuous Integration more than a build that takes a long time. Here I must admit a certain crotchety old guy amusement at what's considered to be a long build. Most of my colleagues consider a build that takes an hour to be totally unreasonable. I remember teams dreaming that they could get it so fast - and occasionally we still run into cases where it's very hard to get builds to that speed.

For most projects, however, the XP guideline of a ten minute build is perfectly within reason. Most of our modern projects achieve this. It's worth putting in concentrated effort to make it happen, because every minute chiseled off the build time is a minute saved for each developer every time they commit. Since Continuous Integration demands frequent commits, this adds up to a lot of the time.

If we're staring at a one hour build time, then getting to a faster build may seem like a daunting prospect. It can even be daunting to work on a new project and think about how to keep things fast. For enterprise applications, at least, we've found the usual bottleneck is testing - particularly tests that involve external services such as a database.

Probably the most crucial step is to start working on setting up a Deployment Pipeline. The idea behind a deployment pipeline (also known as build pipeline or staged build) is that there are in fact multiple builds done in sequence. The commit to the mainline triggers the first build - what I call the commit build. The commit build is the build that's needed when someone pushes commits to the mainline. The commit build is the one that has to be done quickly, as a result it will take a number of shortcuts that will reduce the ability to detect bugs. The trick is to balance the needs of bug finding and speed so that a good commit build is stable enough for other people to work on.

Once the commit build is good then other people can work on the code with confidence. However there are further, slower, tests that we can start to do. Additional machines can run further testing routines on the build that take longer to do.

A simple example of this is a two stage deployment pipeline. The first stage would do the compilation and run tests that are more localized unit tests with slow services replaced by Test Doubles, such as a fake in-memory database or a stub for an external service. Such tests can run very fast, keeping within the ten minute guideline. However any bugs that involve larger scale interactions, particularly those involving the real database, won't be found. The second stage build runs a different suite of tests that do hit a real database and involve more end-to-end behavior. This suite might take a couple of hours to run.

In this scenario people use the first stage as the commit build and use this as their main CI cycle. If the secondary build fails, then this may not have the same 'stop everything' quality, but the team does aim to fix such bugs as rapidly as possible, while keeping the commit build running. Since the secondary build may be much slower, it may not run after every commit. In that case it runs as often as it can, picking the last good build from the commit stage.

If the secondary build detects a bug, that's a sign that the commit build could do with another test. As much as possible we want to ensure that any later-stage failure leads to new tests in the commit build that would have caught the bug, so the bug stays fixed in the commit build. This way the commit tests are strengthened whenever something gets past them. There are cases where there's no way to build a fast-running test that exposes the bug, so we may decide to only test for that condition in the secondary build. Most of the time, fortunately, we can add suitable tests to the commit build.

Another way to speed things up is to use parallelism and multiple machines. Cloud environments, in particular, allow teams to easily spin up a small fleet of servers for builds. Providing the tests can run reasonably independently, which well-written tests can, then using such a fleet can get very rapid build times. Such parallel cloud builds may also be worthwhile to a developer's pre-integration build too.

While we're considering the broader build process, it's worth mentioning another category of automation, interaction with dependencies. Most software uses a large range of dependent software produced by different organizations. Changes in these dependencies can cause breakages in the product. A team should thus automatically check for new versions of dependencies and integrate them into the build, essentially as if they were another team member. This should be done frequently, usually at least daily, depending on the rate of change of the dependencies. A similar approach should be used with running Contract Tests. If these dependency interactions go red, they don't have the same “stop the line” effect as regular build failures, but do require prompt action by the team to investigate and fix.

Hide Work-in-Progress

Continuous Integration means integrating as soon as there is a little forward progress and the build is healthy. Frequently this suggests integrating before a user-visible feature is fully formed and ready for release. We thus need to consider how to deal with latent code: code that's part of an unfinished feature that's present in a live release.

Some people worry about latent code, because it's putting non-production quality code into the released executable. Teams doing Continuous Integration ensure that all code sent to the mainline is production quality, together with the tests that verify the code. Latent code may never be executed in production, but that doesn't stop it from being exercised in tests.

We can prevent the code being executed in production by using a Keystone Interface - ensuring the interface that provides a path to the new feature is the last thing we add to the code base. Tests can still check the code at all levels other than that final interface. In a well-designed system, such interface elements should be minimal and thus simple to add with a short programming episode.

Using Dark Launching we can test some changes in production before we make them visible to the user. This technique is useful for assessing the impact on performance,

Keystones cover most cases of latent code, but for occasions where that's not possible we use Feature Flags. Feature flags are checked whenever we are about to execute latent code, they are set as part of the environment, perhaps in an environment-specific configuration file. That way the latent code can be active for testing, but disabled in production. As well as enabling Continuous Integration, feature flags also make it easier for runtime switching for A/B testing and Canary Releases. We then make sure we remove this logic promptly once a feature is fully released, so that the flags don't clutter the code base.

Branch By Abstraction is another technique for managing latent code, which is particularly useful for large infrastructural changes within a code base. Essentially this creates an internal interface to the modules that are being changed. The interface can then route between old and new logic, gradually replacing execution paths over time. We've seen this done to switch such pervasive elements as changing the persistence platform.

When introducing a new feature, we should always ensure that we can rollback in case of problems. Parallel Change (aka expand-contract) breaks a change into reversible steps. For example, if we rename a database field, we first create a new field with the new name, then write to both old and new fields, then copy data from the exisitng old fields, then read from the new field, and only then remove the old field. We can reverse any of these steps, which would not be possible if we made such a change all at once. Teams using Continuous Integration often look to break up changes in this way, keeping changes small and easy to undo.

Test in a Clone of the Production Environment

The point of testing is to flush out, under controlled conditions, any problem that the system will have in production. A significant part of this is the environment within which the production system will run. If we test in a different environment, every difference results in a risk that what happens under test won't happen in production.

Consequently, we want to set up our test environment to be as exact a mimic of our production environment as possible. Use the same database software, with the same versions, use the same version of the operating system. Put all the appropriate libraries that are in the production environment into the test environment, even if the system doesn't actually use them. Use the same IP addresses and ports, run it on the same hardware.

Virtual environments make it much easier than it was in the past to do this. We run production software in containers, and reliably build exactly the same containers for testing, even in a developer's workspace. It's worth the effort and cost to do this, the price is usually small compared to hunting down a single bug that crawled out of the hole created by environment mismatches.

Some software is designed to run in multiple environments, such as different operating systems and platform versions. The deployment pipeline should arrange for testing in all of these environments in parallel.

A point to take care of is when the production environment isn't as good as the development environment. Will the production software be running on machines connected with dodgy wifi, like smartphones? Then ensure a test environment mimics poor network connections.

Everyone can see what's happening

Continuous Integration is all about communication, so we want to ensure that everyone can easily see the state of the system and the changes that have been made to it.

One of the most important things to communicate is the state of the mainline build. CI Services have dashboards that allow everyone to see the state of any builds they are running. Often they link with other tools to broadcast build information to internal social media tools such as Slack. IDEs often have hooks into these mechanisms, so developers can be alerted while still inside the tool they are using for much of their work. Many teams only send out notifications for build failures, but I think it's worth sending out messages on success too. That way people get used to the regular signals and get a sense for the length of the build. Not to mention the fact that it's nice to get a “well done” every day, even if it's only from a CI server.

Teams that share a physical space often have some kind of always-on physical display for the build. Usually this takes the form of a large screen showing a simplified dashboard. This is particularly valuable to alert everyone to a broken build, often using the red/green colors on the mainline commit build.

One of the older physical displays I rather liked were the use of red and green lava lamps. One of the features of a lava lamp is that after they are turned on for a while they start to bubble. The idea was that if the red lamp came on, the team should fix the build before it starts to bubble. Physical displays for build status often got playful, adding some quirky personality to a team's workspace. I have fond memories of a dancing rabbit.

As well as the current state of the build, these displays can show useful information about recent history, which can be an indicator of project health. Back at the turn of the century I worked with a team who had a history of being unable to create stable builds. We put a calendar on the wall that showed a full year with a small square for each day. Every day the QA group would put a green sticker on the day if they had received one stable build that passed the commit tests, otherwise a red square. Over time the calendar revealed the state of the build process showing a steady improvement until green squares were so common that the calendar disappeared - its purpose fulfilled.

Automate Deployment

To do Continuous Integration we need multiple environments, one to run commit tests, probably more to run further parts of the deployment pipeline. Since we are moving executables between these environments multiple times a day, we'll want to do this automatically. So it's important to have scripts that will allow us to deploy the application into any environment easily.

With modern tools for virtualization, containerization, and serverless we can go further. Not just have scripts to deploy the product, but also scripts to build the required environment from scratch. This way we can start with a bare-bones environment that's available off-the-shelf, create the environment we need for the product to run, install the product, and run it - all entirely automatically. If we're using feature flags to hide work-in-progress, then these environments can be set up with all the feature-flags on, so these features can be tested with all immanent interactions.

A natural consequence of this is that these same scripts allow us to deploy into production with similar ease. Many teams deploy new code into production multiple times a day using these automations, but even if we choose a less frequent cadence, automatic deployment helps speed up the process and reduces errors. It's also a cheap option since it just uses the same capabilities that we use to deploy into test environments.

If we deploy into production automatically, one extra capability we find handy is automated rollback. Bad things do happen from time to time, and if smelly brown substances hit rotating metal, it's good to be able to quickly go back to the last known good state. Being able to automatically revert also reduces a lot of the tension of deployment, encouraging people to deploy more frequently and thus get new features out to users quickly. Blue Green Deployment allows us to both make new versions live quickly, and to roll back equally quickly if needed, by shifting traffic between deployed versions.

Automated Deployment make it easier to set up Canary Releases, deploying a new version of a product to a subset of our users in order to flush out problems before releasing to the full population.

Mobile applications are good examples of where it's essential to automate deployment into test environments, in this case onto devices so that a new version can be explored before invoking the guardians of the App Store. Indeed any device-bound software needs ways to easily get new versions on to test devices.

When deploying software like this, remember to ensure that version information is visible. An about screen should contain a build id that ties back to version control, logs should make it easy to see which version of the software is running, there should be some API endpoint that will give version information.

Reduced risk of delivery delays

It's very hard to estimate how long it takes to do a complex integration. Sometimes it can be a struggle to merge in git, but then all works well. Other times it can be a quick merge, but a subtle integration bug takes days to find and fix. The longer the time between integrations, the more code to integrate, the longer it takes - but what's worse is the increase in unpredictability.

This all makes pre-release integration a special form of nightmare. Because the integration is one of the last steps before release, time is already tight and the pressure is on. Having a hard-to-predict phase late in the day means we have a significant risk that's very difficult to mitigate. That was why my 80's memory is so strong, and it's hardly the only time I've seen projects stuck in an integration hell, where every time they fix an integration bug, two more pop up.

Any steps to increase integration frequency lowers this risk. The less integration there is to do, the less unknown time there is before a new release is ready. Feature Branching helps by pushing this integration work to individual feature streams, so that, if left alone, a stream can push to mainline as soon as the feature is ready.

But that left alone point is important. If anyone else pushes to mainline, then we introduce some integration work before the feature is done. Because the branches are isolated, a developer working on one branch doesn't have much visibility about what other features may push, and how much work would be involved to integrate them. While there is a danger that high priority features can face integration delays, we can manage this by preventing pushes of lower-priority features.

Continuous Integration effectively eliminates delivery risk. The integrations are so small that they usually proceed without comment. An awkward integration would be one that takes more than a few minutes to resolve. The very worst case would be conflict that causes someone to restart their work from scratch, but that would still be less than a day's work to lose, and is thus not going to be something that's likely to trouble a board of stakeholders. Furthermore we're doing integration regularly as we develop the software, so we can face problems while we have more time to deal with them and can practice how to resolve them.

Even if a team isn't releasing to production regularly, Continuous Integration is important because it allows everyone to see exactly what the state of the product is. There's no hidden integration efforts that need to be done before release, any effort in integration is already baked in.

Less time wasted in integration

I've not seen any serious studies that measure how time spent on integration matches the size of integrations, but my anecdotal evidence strongly suggests that the relationship isn't linear. If there's twice as much code to integrate, it's more likely to be four times as long to carry out the integration. It's rather like how we need three lines to fully connect three nodes, but six lines to connect four of them. Integration is all about connections, hence the non-linear increase, one that's reflected in the experience of my colleagues.

In organizations that are using feature branches, much of this lost time is felt by the individual. Several hours spent trying to rebase on a big change to mainline is frustrating. A few days spent waiting for a code review on a finished pull request, which another big mainline change during the waiting period is even more frustrating. Having to put work on a new feature aside to debug a problem found in an integration test of feature finished two weeks ago saps productivity.

When we're doing Continuous Integration, integration is generally a non-event. I pull down the mainline, run the build, and push. If there is a conflict, the small amount of code I've written is fresh in my mind, so it's usually easy to see. The workflow is regular, so we're practiced at it, and we're incentives to automate it as much as possible.

Like many of these non-linear effects, integration can easily become a trap where people learn the wrong lesson. A difficult integration may be so traumatic that a team decides it should do integrations less often, which only exacerbates the problem in the future.

What's happening here is that we are seeing much closer collaboration between the members of the team. Should two developers make decisions that conflict, we find out when we integrate. So the less time between integrations, the less time before we detect the conflict, and we can deal with the conflict before it grows too big. With high-frequency integration, our source control system becomes a communication channel, one that can communicate things that can otherwise be unsaid.

Less Bugs

Bugs - these are the nasty things that destroy confidence and mess up schedules and reputations. Bugs in deployed software make users angry with us. Bugs cropping up during regular development get in our way, making it harder to get the rest of the software working correctly.

Continuous Integration doesn't get rid of bugs, but it does make them dramatically easier to find and remove. This is less because of the high-frequency integration and more due to the essential introduction of self-testing code. Continuous Integration doesn't work without self-testing code because without decent tests, we can't keep a healthy mainline. Continuous Integration thus institutes a regular regimen of testing. If the tests are inadequate, the team will quickly notice, and can take corrective action. If a bug appears due to a semantic conflict, it's easy to detect because there's only a small amount of code to be integrated. Frequent integrations also work well with Diff Debugging, so even a bug noticed weeks later can be narrowed down to a small change.

Bugs are also cumulative. The more bugs we have, the harder it is to remove each one. This is partly because we get bug interactions, where failures show as the result of multiple faults - making each fault harder to find. It's also psychological - people have less energy to find and get rid of bugs when there are many of them. Thus self-testing code reinforced by Continuous Integration has another exponential effect in reducing the problems caused by defects.

This runs into another phenomenon that many people find counter-intuitive. Seeing how often introducing a change means introducing bugs, people conclude that to have high reliability software they need to slow down the release rate. This was firmly contradicted by the DORA research program led by Nicole Forsgren. They found that elite teams deployed to production more rapidly, more frequently, and had a dramatically lower incidence of failure when they made these changes. The research also finds that teams have higher levels of performance when they have three or fewer active branches in the application’s code repository, merge branches to mainline at least once a day, and don’t have code freezes or integration phases.

Enables Refactoring for sustained productivity

Most teams observe that over time, codebases deteriorate. Early decisions were good at the time, but are no longer optimal after six month's work. But changing the code to incorporate what the team has learned means introducing changes deep in the existing code, which results in difficult merges which are both time-consuming and full of risk. Everyone recalls that time someone made what would be a good change for the future, but caused days of effort breaking other people's work. Given that experience, nobody wants to rework the structure of existing code, even though it's now awkward for everyone to build on, thus slowing down delivery of new features.

Refactoring is an essential technique to attenuate and indeed reverse this process of decay. A team that refactors regularly has a disciplined technique to improve the structure of a code base by using small, behavior-preserving transformations of the code. These characteristics of the transformations greatly reduce their chances of introducing bugs, and they can be done quickly, especially when supported by a foundation of self-testing code. Applying refactoring at every opportunity, a team can improve the structure of an existing codebase, making it easier and faster to add new capabilities.

But this happy story can be torpedoed by integration woes. A two week refactoring session may greatly improve the code, but result in long merges because everyone else has been spending the last two weeks working with the old structure. This raises the costs of refactoring to prohibitive levels. Frequent integration solves this dilemma by ensuring that both those doing the refactoring and everyone else are regularly synchronizing their work. When using Continuous Integration, if someone makes intrusive changes to a core library I'm using, I only have to adjust a few hours of programming to these changes. If they do something that clashes with the direction of my changes, I know right away, so have the opportunity to talk to them so we can figure out a better way forward.

So far in this article I've raised several counter-intuitive notions about the merits of high-frequency integration: that the more often we integrate, the less time we spend integrating, and that frequent integration leads to less bugs. Here is perhaps the most important counter-intuitive notion in software development: that teams that spend a lot of effort keeping their code base healthy deliver features faster and cheaper. Time invested in writing tests and refactoring delivers impressive returns in delivery speed, and Continuous Integration is a core part of making that work in a team setting.

Release to Production is a business decision

Imagine we are demonstrating some newly built feature to a stakeholder, and she reacts by saying - “this is really cool, and would make a big business impact. How long before we can make this live?” If that feature is being shown on an unintegrated branch, then the answer may be weeks or months, particularly if there is poor automation on the path to production. Continuous Integration allows us to maintain a Release-Ready Mainline, which means the decision to release the latest version of the product into production is purely a business decision. If the stakeholders want the latest to go live, it's a matter of minutes running an automated pipeline to make it so. This allows the customers of the software greater control of when features are released, and encourages them to collaborate more closely with the development team

Continuous Integration and a Release-Ready Mainline removes one of the biggest barriers to frequent deployment. Frequent deployment is valuable because it allows our users to get new features more rapidly, to give more rapid feedback on those features, and generally become more collaborative in the development cycle. This helps break down the barriers between customers and development - barriers which I believe are the biggest barriers to successful software development.

Where did Continuous Integration come from?

Continuous Integration was developed as a practice by Kent Beck as part of Extreme Programming in the 1990s. At that time pre-release integration was the norm, with release frequencies often measured in years. There had been a general push to iterative development, with faster release cycles. But few teams were thinking in weeks between releases. Kent defined the practice, developed it with projects he worked on, and established how it interacted with the other key practices upon which it relies.

Microsoft had been known for doing daily builds (usually overnight), but without the testing regimen or the focus on fixing defects that are such crucial elements of Continuous Integration.

Some people credit Grady Booch for coining the term, but he only used the phrase as an offhand description in a single sentence in his object-oriented design book. He did not treat it as a defined practice, indeed it didn't appear in the index.

What is the difference between Continuous Integration and Trunk-Based Development?

As CI Services became popular, many people used them to run regular builds on feature branches. This, as explained above, isn't Continuous Integration at all, but it led to many people saying (and thinking) they were doing Continuous Integration when they were doing something significantly different, which causes a lot of confusion.

Some folks decided to tackle this Semantic Diffusion by coining a new term: Trunk-Based Development. In general I see this as a synonym to Continuous Integration and acknowledge that it doesn't tend to suffer from confusion with “running Jenkins on our feature branches”. I've read some people trying to formulate some distinction between the two, but I find these distinctions are neither consistent nor compelling.

I don't use the term Trunk-Based Development, partly because I don't think coining a new name is a good way to counter semantic diffusion, but mostly because renaming this technique rudely erases the work of those, especially Kent Beck, who championed and developed Continuous Integration in the beginning.

Despite me avoiding the term, there is a lot of good information about Continuous Integration that's written under the flag of Trunk-Based Development. In particular, Paul Hammant has written a lot of excellent material on his website.

Can we run a CI Service on our feature branches?

The simple answer is “yes - but you're not doing Continuous Integration”. The key principle here is that “Everyone Commits To the Mainline Every Day”. Doing an automated build on feature branches is useful, but it is only semi-integration.

However it is a common confusion that using a daemon build in this way is what Continuous Integration is about. The confusion comes from calling these tools Continuous Integration Services, a better term would be something like “Continuous Build Services”. While using a CI Service is a useful aid to doing Continuous Integration, we shouldn't confuse a tool for the practice.

Can a team do both Continuous Integration and Feature Branching at the same time?

In general, Continuous Integration and Feature Branching are mutually exclusive approaches. Most folks who think they are doing both are running a CI Service on their Feature Branches, which as I explain in the previous question, isn't doing Continuous Integration.

There is one situation where it is possible to do both, that is when all the features are so small they can be completed in less than a day. But that seems to be a very rare case, and most people would just call that Continuous Integration.

A secondary point here is that it's perfectly permissible to do personal work on a separate branch, then merge it with main and push when I integrate. I might do that if I were worried I'd fat-finger my IDE and push a broken local main by accident. The key question is whether I'm integrating continuously, not how I manage my personal workspace.

What is the difference between Continuous Integration and Continuous Delivery?

The early descriptions of Continuous Integration focused on the cycle of developer integration with the mainline in the team's development environment. Such descriptions didn't talk much about the journey from an integrated mainline to a production release. That doesn't mean they weren't in people's minds. Practices like “Automate Deployment” and “Test in a Clone of the Production Environment” clearly indicate a recognition of the path to production.

In some situations, there wasn't much else after mainline integration. I recall Kent showing me a system he was working on in Switzerland in the late 90's where they deployed to production, every day, automatically. But this was a Smalltalk system, that didn't have complicated steps for a production deploy. In the early 2000s at Thoughtworks, we often had situations where that path to production was much more complicated. This led to the notion that there was an activity beyond Continuous Integration that addressed that path. That activity came to knows as Continuous Delivery.

The aim of Continuous Delivery is that the product should always be in a state where we can release the latest build. This is essentially ensuring that the release to production is a business decision.

For many people these days, Continuous Integration is about integrating code to the mainline in the development team's environment, and Continuous Delivery is the rest of the deployment pipeline heading to a production release. Some people treat Continuous Delivery as encompassing Continuous Integration, others see them as closely linked partners, often with the moniker CI/CD. Others argue that Continuous Delivery is merely a synonym for Continuous Integration.

How does Continuous Deployment fit in with all this?

Continuous Integration ensures everyone integrates their code at least daily to the mainline in version control. Continuous Delivery then carries out any steps required to ensure that the product is releasable to product whenever anyone wishes. Continuous Deployment means the product is automatically released to production whenever it passes all the automated tests in the deployment pipeline.

With Continuous Deployment every commit pushed to mainline as part of Continuous Integration will be automatically deployed to production providing all of the verifications in the deployment pipeline are green. Continuous Delivery just assures that this is possible (and is thus a pre-requisite for Continuous Deployment).

How do we do pull requests and code reviews?

Pull Requests, an artifact of GitHub, are now widely used on software projects. Essentially they provide a way to add some process to the push to mainline, usually involving a pre-integration code review, requiring another developer to approve before the push can be accepted into the mainline. They developed mostly in the context of feature branching in open-source projects, ensuring that the maintainers of a project can review that a contribution fits properly into the style and future intentions of the project.

The pre-integration code review can be problematic for Continuous Integration because it usually adds significant friction to the integration process. Instead of an automated process that can be done within minutes, we have to find someone to do the code review, schedule their time, and wait for feedback before the review is accepted. Although some organizations may be able to get to flow within minutes, this can easily end up being hours or days - breaking the timing that makes Continuous Integration work.

Those who do Continuous Integration deal with this by reframing how code review fits into their workflow. Pair Programming is popular because it creates a continuous real-time code review as the code is being written, producing a much faster feedback loop for the review. The Ship / Show / Ask process encourages teams to use a blocking code review only when necessary, recognizing that post-integration review is often a better bet as it doesn't interfere with integration frequency. Many teams find that Refinement Code Review is an important force to maintaining a healthy code base, but works at its best when Continuous Integration produces an environment friendly to refactoring.

We should remember that pre-integration review grew out of an open-source context where contributions appear impromptu from weakly connected developers. Practices that are effective in that environment need to be reassessed for a full-time team of closely-knit staff.

How do we handle databases?

Databases offer a specific challenge as we increase integration frequency. It's easy to include database schema definitions and load scripts for test data in the version-controlled sources. But that doesn't help us with data outside of version-control, such as production databases. If we change the database schema, we need to know how to handle existing data.

With traditional pre-release integration, data migration is a considerable challenge, often spinning up special teams just to carry out the migration. At first blush, attempting high-frequency integration would introduce an untenable amount of data migration work.

In practice, however, a change in perspective removes this problem. We faced this issue in Thoughtworks on our early projects using Continuous Integration, and solved it by shifting to an Evolutionary Database Design approach, developed by my colleague Pramod Sadalage. The key to this methodology is to define database schema and data through a series of migration scripts, that alter both the database schema and data. Each migration is small, so is easy to reason about and test. The migrations compose naturally, so we can run hundreds of migrations in sequence to perform significant schema changes and migrate the data as we go. We can store these migrations in version-control in sync with the data access code in the application, allowing us to build any version of the software, with the correct schema and correctly structured data. These migrations can be run on test data, and on production databases.