Friday, February 3, 2012

A Tale of Two Burn Charts

Within the last month we've launched two teams at Walmart Labs using methodology tuning as our means of incrementally introducing Agile. The new "process" introduced for each team was more or less the same:

1) Team will work in iterations
2) Team will create tasks from the user stories allocated to the iteration, and provide work estimates for tasks based on a simple time scale - 2 hours, 4 hours, 1 day. 
4) Tasks are defined as "done" when code/assets/other is checked in and any acceptance criteria for the task has been validated. Iteration is "done" when the working code is deployed to stakeholders and QA has signed off on all tasks/stories within the iteration.
5) QA will work directly with the team, recording and completing their tasks in the same manner as the rest of the team in addition to acceptance evaluation as needed on other tasks.

A few differences of note between the teams: 

1) Team A opted to try out 3 week iterations, Team B went for 2
2) Team A had an initial training session on iteration planning separate from the actual iteration planning exercise, Team B had training (hastily) incorporated into the iteration planning.
3) Team A was primarily on-site, Team B was widely distributed.

For each team the source of the stories for the iteration were derived from existing sources, namely feature lists not articulated in typical story form. This was not an oversight, the teams did not have shared experience in working with Agile style stories, so rather than impose additional training burden on them we collectively agreed to work with the existing resources and revisit the story issue at the next reflection session.

One of the specific "pains" that the overall group was suffering was a lack of visibility into what was happening in the midst of development. This manifest itself in predictable ways - release schedule slips, lack of stakeholder knowledge of feature changes, etc. In addition to the methodology changes that the team agreed to listed above I made it a point to generate daily burn charts on the tasks the teams were performing during the iteration.

Rather than provide a day-by-day running commentary of the iteration, let's skip to the end to see if the butler really did do it:

Team A Task Burn Chart:
Red line denotes ideal burn rate for the iteration
Green line denotes actual team completion of tasks
Blue line on bottom of graph denotes new tasks introduced during the iteration

Team B Task Burn Chart:
Red line denotes ideal burn rate for the iteration
Green line denotes actual team completion of tasks
Blue line on bottom of graph denotes new tasks introduced during the iteration

Based on the burn charts alone, guess which team had delivered working software at the end of their iteration. Go ahead - I'll give you a minute to make your guess.


Ready? It was neither. 

Surprised? Don't be. Burn charts are a tool to understand what is happening during the course of an iteration, not a predictor of success. Even with experienced teams that have a track record of successful delivery each iteration it is entirely possible to fail to deliver at the end of an iteration. For teams that are using burn charts for the first time the burn charts serve only one useful purpose, which is to raise awareness to the team on what sort of underlying problems a burn chart can indicate once a team has some history of delivering in iterations.

Let's take a little walking tour of what occurred during each iteration for the two teams:

Stop #1 (Team A and B)
Note that early in the iteration both teams discovered a number of tasks not previously captured. For Team A, it was 5 new tasks, Team B added 12. Considering that the purpose of the iteration is to provide a period of focus for the team by not allowing changes to the work during that time, adding new tasks certainly seems to indicate that new features showed up during the iteration. 

What actually happened was that both teams realized that there were tasks that were missing from the stories that they had agreed to deliver and as a result they added the new tasks to the iteration without realizing that they had invalidated their earlier work estimates for the iteration and increased risk that they would not deliver on time.

Stop #2 (Team B)
After an initial addition of 12 tasks Team B continued adding tasks along the way, peaking up to 4 new tasks in a day twice. What is interesting about this influx of new tasks is that the team didn't detect this problem until it showed up in the burn chart, even though they had already diagnosed it in the initial "methodology building" reflection session. 

Stop #3 (Team B)
Notice that throughout the burn chart there were only three days where the number of net tasks declined. A casual observer might interpret this as "the team isn't doing anything", but that wasn't the case here. Two factors were behind this seeming lack of progress. First, the lead developer (who had taken primary responsibility for managing task state within the tracking tool) was on vacation for a week. In his absence the rest of the team continued to get work done, but didn't update their progress in the tracking tool.

Stop #4 (Team A)
If you look close to the end of the burn chart for Team A, you'll notice a small spike in new tasks two days before the end of the iteration. In this case the tasks in question were not likely to cause a delay in the completion of the iteration but they did represent a lack of understanding of the importance of focus during the iteration. Much like Team B on Stop #2, Team A had not yet developed an aversion to changing work during the iteration.

Stop #5 (Team A)
Look at the end of the burn chart. Notice how it concludes with 4 tasks remaining open? In this case this wasn't a reporting oversight, there were 4 actual tasks that were not completed at the end of the iteration. All 4 of these tasks were work needing to be performed by QA. It would be easy to claim that QA lags behind development and accept that the iteration is complete, but one of the rules of our newly minted methodology was that QA would be performing their work alongside the other team roles, not trailing it. 

End of Tour
Even though there was a huge disparity in the perceived progress of the two teams, the reality is that both teams finished the development work more or less on time. Team A had less overall work remaining to complete their iteration because of a closer communication between Dev and QA. Team B had more work to complete the iteration because QA was not as aware of progress and started their work later. Team B also had to spend time correlating completed tasks with their relevant stories (OK, feature lists) to communicate to stakeholders what user-centric functionality had been completed in the iteration.

The moral of our story is that neither team failed - both executed against their tasks rather well, at least from the development perspective. Visibility definitely improved, as did concrete visibility for both teams of previously hidden inefficiencies (effect of late QA involvement, lack of visibility of progress). Did the achieve total success in all of their corrective actions? No, nor was there ever an expectation that they should. The teams did learn something useful from the changes in their work and have already applied this knowledge to their current iterations of work to continue the process of improvement.

No comments:

Post a Comment