Agile: Monitoring the Health of Projects

Paper readout from an EKG

Monitoring the Health of Agile Projects: Choosing the Right Data

Profile iconDoug Vahlkamp, Adam Monsalve

Waterfall software development defines schedule, budget, and scope prior to the start of development. This allows for the use of traditional metrics (such as those derived from an integrated master schedule or overall project workflow) to measure the health of a waterfall project. Agile software development is often non-linear—the embrace of changing requirements prevents the use of many traditional project health metrics that rely on a predefined scope to drive schedule and cost estimates. Agile requires unique metrics to measure project health, and understanding these metrics early-on is critical to ensuring that the project is monitoring the necessary parameters.

We reviewed a variety of agile metrics (and the products used to visualize them) and mapped them to the project management needs in the table below. The project manager can use this information to inform project status tracking and ensure the necessary parameters are being measured. For example, if the project has strict short-term deadlines, an iteration burndown chart can be used to determine if the development team is on-track to complete the committed work for an iteration. If the project manager chooses to use burndown charts, then they should require the development team to regularly provide information on the status of committed work (e.g., tasks or story points completed).

Table 1: Project management needs and the agile metrics that can be used to evaluate them
Metric or product Schedule forecast Work completed Time to complete tasks Budget forecast
Burndown chart
Velocity chart
Cumulative flow diagram
Control chart
Defect trend chart
Indicator for Schedule Analysis and Cost (ISAAC)
Scope change chart

Based on our experience supporting federal agile development projects, CNA recommends using burndown charts and velocity charts to monitor project health. In addition, CNA has introduced the Indicator for Schedule Analysis and Cost (ISAAC) tool, which complements traditional Earned Value Management (EVM) metrics and has demonstrated success forecasting agile development project spending and schedule. Used together, these three metrics inform all of the project management needs in Table 1. In the sections below we provide a primer on how each of these metrics can be used and what inputs are required.

Burndown chart

An iteration burndown chart shows the number of story points completed within a single development iteration. It is useful in determining whether an iteration is on schedule, but relies on estimating story points for the features grouped in an iteration. If the development team falls behind during an iteration, this can be an indicator of the capacity of the development team (or the accuracy of story point estimates), and may necessitate updating schedule projections for the remainder of the project. This chart can project the endpoint at which all current story points of that iteration will be complete. It can also assist in monitoring the iteration’s velocity—the speed at which story points are being completed within the iteration. Burndown charts can present the status of the iteration, setting stakeholder expectations of the planned functionality that will be delivered at the end of the iteration.

Metrics burndown chart
Burndown charts offer visual representations of information that stakeholders have been most interested in, particularly as a representation of the status of an iteration

Velocity chart

An iteration velocity chart shows the number of tasks or story points completed within each development iteration. It is often compared against the tasks or story points initially committed to at the start of the iteration. Using this type of chart, the project manager can determine if the appropriate number of story points are being completed each iteration and make adjustments accordingly. Velocity charts are used to better understand the development team’s resources and plan the work capacity for upcoming iterations. This could mean reallocating developers to challenging areas or revising the commitments from each iteration to be more realistic. The average number of tasks or story points completed can also be used to forecast what the development team will be able to accomplish in future iterations.

Metrics velocity chart
Velocity charts help manage resources across iterations, and can be used to forecast whether the project is on track to complete scheduled scope

Indicator for Schedule Analysis and Cost (ISAAC)

EVM metrics compare actual cost against budgeted cost over time and can use that information to forecast the cost and schedule performance at the completion of the project. However, these EVM metrics make several assumptions that often do not hold true for software development projects. For instance, EVM metrics use past behavior to estimate future performance, regardless of the differences in work as the project progresses. In addition, traditional EVM forecasting is highly dependent on the budgeted cost to complete work items—if the effort to complete future work is not well-understood, EVM forecasting can be severely misleading.

CNA has developed its own cost and schedule analysis tool, the Indicator for Schedule Analysis and Cost (ISAAC), to improve upon traditional EVM metrics for IT and software development based projects. This proprietary tool applies a Rayleigh distribution to real project data to forecast final cost and schedule. It works particularly well for forecasting software development projects because it can account for the slow ramp-up of development, peak spending as complexity increases, and gradual transition to operations and maintenance (O&M)—it does not rely on the budgeted estimates for future work used for EVM forecasting. Another benefit of ISAAC is that it can provide early indication of cost and schedule overruns with only a few data points. Since many government software development projects require EVM metrics, ISAAC can provide these unique insights without requiring additional data to be collected. More information about CNA’s ISAAC tool can be found here.

ISAAC image
CNA has developed ISAAC, a schedule and cost forecasting tool, which can provide an early indication of cost and schedule overrun without requiring additional data collection