program selection system

You’re the executive-sponsor of your division’s foray into agile. It was successful with the first development team; you saw good results with the second and third; by the fourth you knew you had something! Now you want to roll it out across the entire organization. But each of your teams approaches agile differently. You’d like a consistent approach, but you don’t want to stifle the culture of cooperation and coordination that agile has brought to the organization. How do you scale with a consistent approach that also supports The Manifesto?

To successfully scale agile across a large development organization, that organization must augment the agile framework with a set of clearly defined, non-negotiable rules and regulations, practices and expectations, and the organization must train itself to constantly and consistently enforce, and question, these conventions from within.

Agilists place a lot of emphasis on the discipline of individuals and the discipline of teams. But this is not enough to scale agile across a large development group. Frameworks, like agile, provide a preferred approach. The organization itself has to commit to a consistent approach.

VeriSign Enterprise Security discovered, when we tried to scale agile across our division, that we required a broader brand of discipline. So we set up non-negotiable practices, procedures, metrics and reporting. We found that when we conformed to these rules we set down we were consistently successful. We also found that when we deviated from our organizational constitution, or when we tried something new without setting down ground rules for everyone to follow, we consistently failed. We found that in addition to setting down rules and policing ourselves, we needed to regularly examine, adjust and communicate out new policies as we learned how we could best leverage agile disciplines to build and deploy successful products.

Deciding to Scale

VeriSign Enterprise Security began implementing agile using scrum in June 2006. In February 2007, after eight months of experimenting with agile development, the division was ready to roll the agile framework out to the entire development organization of more than 40 people in four locations. We had seen demonstrable success with the three teams what were using agile; now we would have eight teams comprised of the entire organization. Our individual engineers and QA people were dedicated to the discipline of agile. They were forming into teams that were dedicated to the practices of agile.

There was just one problem. Which disciplines and practices, exactly, were we talking about?

We were going to a lot of trainings, a lot of conferences, learning from a number of consultants, and reading a lot of books. We hung posters of the Manifesto on every blank wall in the building so that everyone would understand the fundamentals. But, while the fundamentals were consistent across the information sources who were instructing us about agile, the particulars often were very different.

Some agilists recommended 4-week iterations, some recommended 2 or even 1-week iterations. Some recommended using scrum walls with cards and stickies, some used on-line tools like Rally and ScrumWorks. Some insisted that we would fail miserably unless teams were co-located, others told us our distributed structure would not be an obstacle. Our three scrummasters were advocating and practicing some very diverse “agile” and there was a ton of discussion about what our “best practices” ought to be, or if best practices were best left to each team to decide upon.

After weeks of discussion we decided that we needed internal consistency if we were going to scale agile to the entire organization. So as a team we set up some rules to live by.

We also had some organizational goals in mind. If we were going to set down new rules, we should tailor them to solving some of the problems we were facing as a business unit. We were doing a lot of releases to production every quarter, sometimes as many as three or four major releases, each with its accompanying overhead. We determined that we could solve this problem if we set expectations around our new agile development process. We would merge whatever was ready from branches into trunk and release on a set quarterly schedule, which would be the last Wednesday of every quarter. Once this was locked in and communicated out, it would be only in extreme cases where the business could say they couldn’t wait for the quarterly release to get some new technology out to the market. If this worked, we would have the entire organization marching toward a common goal every quarter: completing the milestones that lead to deployment day. We talked about how we wanted our process to work and, like the benign dictatorship that we are, we wound up with these:

Non-negotiable items: • Release kick-offs, Iteration kick-offs, UATs, retrospectives and daily scrums using standard agendas • Scrum wall (physical story cards, task cards) is primary working record of iteration activity • QA integrated into the team and the sprint • Five, two-week iterations every quarter • Definition of done within 2 week iterations = development, QA sign-off, Product Owner acceptance • No changes to User Stories or additional User Stories during the iteration • When team members “go missing” during an iteration for any reason, we will take a hit to velocity and note it in our documentation • Sign-off is QA acceptance (QA test-plan pass) • Standard spreadsheets for velocity tracking and sharing • All major releases will have a code freeze, regression (3 weeks) and pilot (2 weeks) prior to GA on the last Wednesday of every quarter • All major releases will have an entropy reduction iteration running parallel to regression

We clearly communicated these expectations to scrummasters and technical leads, who in turn made them clear to each agile team. We met with stakeholders at all levels to make our intentions clear and to gain their acceptance. Once we were all informed of organizational expectations, we began the process of rolling agile out to everyone.

Community Policing

That decided, now somebody or everybody had to be a cop.

We knew it wouldn’t work unless we were committed to it as a management team. So from the very top down, for the first few weeks, we monitored each team extremely closely. This wasn’t easy. We were not all entirely convinced that we had chosen the correct non-negotiable practices. Some of us were pretty certain that we’d chosen the wrong practices. But we remained united as a management team under a director who made certain that even if we disagreed, we committed. We also committed to re-examine our policies quarterly.

To assist us with our policing we created a wiki space for our required artifacts. Each wiki page displayed a list of user stories per iteration noting any pertinent assumptions, housed all the story test plans, and recorded the attendance and outcomes of each UAT meeting. We published velocity numbers, backlog items and points, retrospective results, regression plans, noted production release information, design documents, security and technical documents and approvals, UI mockups and any MRDs and FRSs. For the edification of future generations, we included the celebratory haikus created by the Athena team after each successful iteration (“18 points this time/ We’re pretty happy with that!/ Somewhere, a leaf falls”).

The wiki gave everyone visibility into the inner workings of each team with minimal invasiveness. We were able to determine pretty quickly if a team was conforming to the non-negotiables and course-correct if they were not.

Conformity had other advantages as well. Although points-estimating was left up to each individual team with no consistent standard imposed, we could compare teams to each other. We could look at how well, over five iterations every quarter, each team predicted velocities, consumed work, integrated QA, and dealt with disruptions (illness, production outages, blockers). We could roll team metrics up to the organizational level. We wound up with a consistent set of artifacts which became extremely valuable during our annual audit process. It was fairly easy for developers and QA resources to move between teams. But most importantly, we were able to implement a quarterly production release schedule, a goal we had had in mind for a very long time. Product Owners and customers became used to and happy with an extremely predictable major quarterly deployment, and Product Owners have found that they can manage market windows within this schedule.

Interestingly, our greatest failures have occurred during instances where we have not set down widely-communicated rules and expectations. We attempted to drive test-driven development through the scrum teams during our third and fourth agile releases. We failed, though, to set down precisely what our expectations were for our development teams and to police and reward or correct developers based on results. As such, we had very spotty results with TDD, with some teams succeeding to implement it, but most not meeting with the level of success for which we had hoped.

Amendments

Now we look at our process every quarter and decide how we need to amend what we’re doing. We roll up information from retrospectives conducted during the release, add agenda items of topics we’d like to explore, and we closely examine our successes and failures. We’ve added new rules surrounding many issues that have cropped up. We don’t let team members move on to a new story if there are any testing issues remaining on an in-progress story, for instance, as we found that the testing was piling up at the end of iterations. We want teams to document UAT criteria specifically at iteration planning meetings and hang the UAT criteria cards up on the scrum wall. We’ve expanded the requirements for a project sign-off to better set up regression and deployment teams with information they need for builds and testing. We implemented a scrum-of-scrums run by our release management team to identify inter-team dependencies and resolve merge issues before they occur.

We remain extremely committed to policing for the benefit of the organization. It has proven extremely effective to have defined sets of non-negotiable expectations, behaviors and deliverables and to make each member of the organization responsible for making sure we are all adhering to our regulations.

• By documenting up front, as an organization, how every team would operate, we were able to scale agile across the operation with very little confusion. We set clear expectation for development and for the business. We created common metrics, a common schedule, and shared goals and expectations.

• By making everyone aware of what was expected and responsible for delivering, we set up a culture focused on behaviors and deliverables that drove us toward success.

• By constantly assessing what works, we let people know that process change is natural and encouraged, and that we’re committed, as an organization, to a successful practice and product.