Feature Crews: How Microsoft Does It

With the advanced concepts behind us it is time for us to discuss our guidance for branching structures which are optimized for large teams. With the release of Visual Studio “Orcas,” Microsoft’s Developer Division went to a feature crew model. This process was originated by the Microsoft Office team and has proven successful across the company. The feature crew model and corresponding feature branches will be the basis for our guidance in this section.

The adoption of the feature crew model within Microsoft’s teams is informed by valuable lessons from past shipping experiences in regards to:

• Inability to reliably predict the ship date of a product because, invariably, the stabilization of features was deferred towards the end of the project lifecycle. This made it impossible to predict how long the stabilization would last before the product could be deemed ship-ready.
• Inability to reduce the scope of features to hit a release date because it was difficult to scale back on features which, although had been coded, had not been stabilized. Scaling back on scope was further exacerbated by the fact that newly developed features had introduced other dependencies around them that were hard to back out. Additionally many of those features had already been shown to customers via product demos and there were certain expectations set about them shipping in the official release.
• Inefficiencies due to the significantly long time between when the feature was developed and when it was fully tested.

The lessons learned from the prior shipping experiences shaped the key guiding principles behind the feature crew model. These principles are:

• Driving quality upstream – ensuring efficiency by requiring teams to meet certain quality gates and to stabilize their code before their feature can be merged back into MAIN.
• Preventing deferred work – ensuring work up front to facilitate early bug detection and resolution, rather than deferring it until later in the development phase
• Keeping the MAIN branch customer-ready which allows for more frequent milestone deliveries to customers without the risk of shipping unstable or incomplete feature code, as well as reducing the time required to prepare the final release.
• Enforcing a common definition of what completion means across various teams via a common, consistent definition of quality gates with meaningful metrics on project progress.

Feature Crew and Feature Defined

A feature crew is a small interdisciplinary team responsible for producing a feature. A feature is an independently testable unit of work that is either directly visible to a customer or is infrastructure that is going to be consumed by another feature. Features should be small enough that they can be worked on by a feature crew – but large enough that it actually makes sense to test it independently.

Features are defined with a goal to optimize for productivity via short cycles – typically 3-6 weeks - of implementation, stabilization and integration into MAIN. This ensures a steady flow of features into MAIN and enables shipping of frequent Customer Technology Previews (CTPs) for valuable, early customer feedback throughout the lifecycle.

A feature should also be designed with a clear understanding and explicit statement of other feature dependencies that will need to have been checked into MAIN before the given feature can be developed. So it helps to break down big features into smaller, manageable deliveries that are independently testable, and prevents destabilization of the product which may otherwise occur if the features are delivered as a big, end-to-end endeavor.

Team Composition & Values:

A typical feature crew is a program manager, five or less developers, and a corresponding number of testers that will work together for 3-6 weeks as a closely knit team with the common goal of hitting feature complete (as defined below).

• Everyone in the feature crew works together within the same timeframe.
• Developers and testers work in tandem – code gets tested as it is developed, so we minimize the lag time between feature development and its quality validation. Feature development happens in feature branches which provide the code isolation needed for the feature crew.
• A feature is “Feature Complete” when it has been fully implemented, is stable, is well tested, and it is either ready for others to consume. This consumption may be in the form of internal testing (aka “dogfooding”), by sharing with external customers in a CTP, or for infrastructure features it is ready to be coded against by another feature crew.

It is easy to see the benefits realized by the feature crew model at Microsoft. It provides teams with a sense of empowerment and collaborative spirit by working on a well-defined, focused goal. The feature crew model enables faster turnaround and iterations between developers, testers, and program managers. It also enables sharing of code between different teams without risking the destabilization of the MAIN branch. Individually, too, the developer division observed improvements in morale due to the focused and collaborative environment which teams worked in.

While most of the best practice guidelines we have discussed with respect to branching and merging still apply in the feature crew model, it helps to summarize and underscore the following key engineering approaches that make the approach unique and effective:

• Drive to Feature Complete, not simply Code Complete. Finish a feature before adding it to the product - spec, design, implement, automate, test and fix bugs
• Use Quality Gates to provide consistency across the division. Define the divisional requirements for Feature Complete. These Quality Gates must be satisfied before adding a feature to the product
• Use Feature Branches to isolate new feature development. Each feature gets its own branch.
• Only Reverse Integrate the feature branch into the PU branch (covered below) once each of the following is met:
  • The feature is complete
  • The Quality Gates are satisfied

One additional characteristic of the Feature Crew model is the introduction of a PU (Product Unit) branch as an intermediate staging branch between the Feature Branch and MAIN. As such, the promotion path for merging now becomes:

Feature Branch <=> PU Staging branch <=> Main, after satisfying a set of Quality Gates. These Quality Gates are designed to prevent work from being deferred whilst striking a sensible balance between the work that we know should be done upfront and the work that can only realistically happen later in the product cycle. Some examples of Quality Gates enforced at Microsoft include Functional Specification, Dev Design, Test Plan, Threat Mode, and IP Protection Code Coverage just to name a few.

The following diagram represents the physical structure of Feature Crew branching.