See Large Scale Multi-Stage Continuous Integration in Real Time
With Continuous Integration, the chances rise exponentially with team size that as you fix a build or test problem, the integration of somebody else’s work creates a new build or test problem. Multi Stage CI addresses this problem by extending the practice of temporary individual isolation to features, teams, staging, QA, and release. This demonstration will show how Multi-Stage Continuous Integration works in the real world by drawing on the experiences of three large distributed teams. See changes by hundreds of simulated developers make their way through the system.
For any meaningful project, a full integration build and test can take 30 minutes during which time other people may check in their changes. This can create a ping-pong effect in which the Continuous Integration system is constantly telling you that something is broken. As developers we shield each other from instability by only checking in when we feel that our changes are integrated, tested, and won’t disrupt other people. Also, we only update our workspace when we are at a point that we feel we are ready to absorb other people’s changes. Multi-Stage Continuous Integration is an extension of this practice. If temporary individual isolation is a good idea, then temporary isolation for features, teams, team integration, staging, QA and release is an even better idea.
As soon as a developer is ready to check-in, instead of checking into the mainline, they check in to their team branch which triggers a normal CI cycle. If that succeeds, then that branch and the mainline are automatically cross-integrated and another CI is triggered for the mainline. Unlike the traditional use of branches for long-lived variants, in Multi-Stage CI, branches act as temporary buffers and changes linger only as long as necessary, similar to the flow associated with a developer workspace. The result is that the impact of a build breaking or test breaking change is usually limited to just a single team instead of the whole world… with zero changes to a developer’s day to day activities.
We will do a very short presentation on the concepts followed by a demonstration of a day in the life of Multi-Stage CI. See changes by hundreds of simulated developers for tens of teams get integrated at the team level, get validated by CI and either automatically move to the mainline or get rejected. This will be followed by CI at the mainline. Also, CI for multiple teams and the mainline will all be happening in parallel. This will be coordinated using Cruise Control, AccuRev, and Jira as an example implementation.
A common reaction after folks move from CI to Multi-Stage CI is: “I never realized how many problems were a result of doing mainline development until they disappeared.”