A Hundred Days of Continuous Integration
Description for Program Guide
Many agile teams use Continuous Integration (CI). It’s one of the XP practices that has been broadly adopted. How effective is it? Does the effort of maintaining the CI server save time over a lengthier process that attempts to never break the build? While much anecdotal evidence exists to the benefits of CI there is little supporting data. How do you convince teams that it’s worth adopting and how best to do it? This report covers experiences with CI in a distributed team environment and attempts to answer these questions.
Paper Abstract
Many agile teams use Continuous Integration (CI). It is one of the Extreme Programming practices and has been broadly adopted by the community [1]. Just how effective is it? Does the effort of maintaining the CI server and fixing build breaks save time compared to a lengthier check-in process that attempts to never break the build? While much anecdotal evidence exists as to the benefits of CI there is very little in the way of data to support this. How do you convince teams and management that it’s worth adopting and how best to do it? This report outlines our experience with CI in a distributed team environment and attempts to answer these questions.
Original Proposal
This report outlines our experiences with CI a distributed team of developers over several months. It uses data from the CI server and source code control system to evaluate its effectiveness. The data allowed us to see the main causes of build breaks and their impact and frequency. These metrics can be used as one measure of team health.
We used the same data to calculate the amount of time spent maintaining and fixing the build. We then compared this to a model based on the same data that assumed a more conventional, lengthy “get check-ins right first time” approach. The CI based approach shows significant time savings.
We also consider the pros and cons of other check-in strategies and their potential cost. Our experience includes some lessons learnt and best practices we would recommend when using CI on a distributed team.
A draft of this paper has been circulated internally at Microsoft and the feedback has been very positive. I would welcome feedback on this and the opportunity to present this to a wider audience. I can make the draft available to the review committee.
Answers to specific questions raised by reviewers:
Kirk Knoernschild - If I make the paper I have now available prior to the conference the IEEE considers it already published and therefore ineligible for conference publication (at least that’s my understanding). This talk is definitely an experience report, hence the content and length. I am considering submitting another clinic/talk on CI best practices.
Fabrizio Cannizzo – I think there will definitely be some variation in the effectiveness of CI depending on the product under development and the composition and distribution of the team. I could mitigate this by adding additional data from one of our other projects although this would probably be considered similar in terms of team composition and distribution.
This will be a 25 minute talk with the remaining 5 minutes for questions. I’m suggesting this as it is the same format as experience reports at Agile 2007. I’d be happy to talk for slightly longer if this would have merit.
I will give a very brief overview of Continuous Integration (CI) and give the audience some context in terms of the distributed makeup of our team and the sort of project we were working on. We’ll then review some of the data analysis; what broke the build, for how long and how often. This leads to some interesting results around the value of unit testing and static analysis as part of a CI process. These results are applicable to all teams not just distributed ones.
Next I’ll present the amount of time spent maintaining and fixing the CI server based on the data and then compare this with my (“traditional”) model in which developers try and never break the build to show the expected savings. I’ll also consider other models and their attendent limitations, e.g. modifying the “traditional” model by checking in less frequently to reduce checkin overhead.
Finally I’ll cover some of the best practices around CI we used on our team and summarize my findings overall.
Note: Currently this is listed in the Developer Jam stage but I’m not entirely clear where this fits best; Committing to Quality and possibly Questioning Agile also seem to have some sort of fit.