“There have been many attempts to measure the performance of software teams. Most of these measurements focus on productivity. In general, they suffer from two drawbacks. First, they focus on outputs rather than outcomes. Second, they focus on individual or local measures rather than team or global ones.” 1
PMO and IT Finance reports are distributed through the reporting lines and are, therefore, prepared for the people on the distribution list of the specific report. Unlike PMO and IT Finance type reports, audience of the Lean-Agile Dashboards would be Agile teams, all internal stakeholders including senior management, and IT Finance as well as wider audience in the organisation who want or need to learn from experience and be part of the innovation and transformation.
However, specific dashboards help to lead action for specific group of audience more than others. Therefore, the Digital Dashboards are grouped by the main audience and subject category of the metrics.
Value and Growth Dashboard
Value and Growth Dashboard is the dashboard where key outcome metrics of the product are collected. The main audience of this dashboard would be Senior Management and other relevant internal stakeholders as well as Agile Teams.
However, unlike the other three dashboards that will be explained later in this article, I will not suggest any metrics for this specific dashboard because metrics on Value and Growth Dashboard vary depending on the organization and product stages. “You can’t just start measuring everything at once. You have to measure your assumptions in the right order. To do that, you need to know what stage you are at.” 2 The exploit or explore stage of the product requires different metrics. Even the explore stage for a start-up product 3 requires focusing on different metrics at different sub-stages of the product life-cycle. 4
Again, similarly, you cannot use the same Value and Growth metrics for different business models, such as Mobile App, eCommerce and Media websites. They all have different business goals related to their business model. Therefore, despite having some common metrics, they would also require focusing on different metrics related to the business model. For example, whereas Conversion Rate is the one of the key metrics for eCommerce business, Subscription Rate would be a key metric for SaaS (Software as a Service) business. Your metrics for Value and Growth tightly bind your product stage and type.
Nevertheless, it’s key to have Value and Growth Dashboard as a foundation of the Lean-Agile Dashboards. First of all, we measure success by outcomes in a Lean-Agile product development environment. If we build the other three dashboards (Agility, Quality and Epic Overview) without the foundation, it might cause a slip of the definition of success for the organization. Let’s assume, some of our key metrics on other dashboards show significant improvement. For example, we have no P1 Bugs/Defects or Failed Test Results on the Quality Dashboard for over the desired period; or we have successfully optimized Average Cycle Time metric on the Agility Dashboard, would it mean the “success” while we are losing 5-10% Conversion Rate against previous period as a result of our product development activities? In other words, as Eric Ries pointed out as a comparison of plan based and Lean-Agile approaches, “If we are busy building the wrong product, why should be proud to be doing it on time and on budget?” This is where we can see whether we are in the right direction. 5
In an enterprise environment transforming towards Lean-Agile mind-set, it’s a process of cultural change via continuous learning and adaptation. One of the biggest challenges as part of this cultural adaptation process is changing the definition of success from a plan-based project development mindset to the Lean-Agile product development mindset. Therefore, Value and GrowthDashboard is not only important for Senior Management and Agile Teams to see whether they are on the right path, it’s also important for the organization to learn and shift the definition of success from an output-based one to an outcome based one. The more they see outcome-based metrics as a criteria for evaluation of success, the more they start seeing “success” from an outcome perspective.
Agility Dashboard
The aim of the Agility Dashboard is to provide visibility over the metrics of Agile Development for the Team of Teams (Tribe) and Teams (Squad). The main audience of Agility Dashboard is Agile teams, therefore, graphs and metrics on this dashboard should include Teams of Teams (Tribe) and Team (Squad) breakdown.
As it’s explained above, dashboards are meant to contain only brief and key metrics, therefore, I will be recommending only a few key metrics for each dashboard below. Also, needless to say, some metrics might not fit with all type of organizations. Therefore, my suggestions below should be taken as examples. Eventually, the organization needs to decide whether they are valuable and key metrics to improve their Lean-Agile development.
Some metrics for Agility Dashboard:
Average Cycle Time: It’s the average time spent for user stories between its status change to Ready for Development and Released. It’s important to reduce cycle time of development of the items so that the team can learn and adapt quickly via shorter feedback loop.
Work Worth Week Ready for Development: Another important target for Scaled Agile development teams is to keep Work in Progress (WIP) within healthy limits. This metric is to show whether we have WIP in a health range. Too few or too many items on team backlog would be a signal of unhealthy management of queue lengths.
WWWRD = Total Story Points of the team backlog items/last six weeks average weekly velocity if team.
Releases: In Continuous Delivery Pipeline, changes are continuously deployed 6 into the production whereas releases are immediately or incrementally to customers on demand 7. Nevertheless, a shorter release cycle is preferred to learn from customers and adjust the product in the right direction. It also helps to expose correlations between releases and other metrics such as pattern of the increasing or decreasing Conversion Rates on the Value and Growth Dashboard or number of call centre calls related to the product on the Quality Dashboard.
Team Velocity: “In standard Scrum, each team’s story point estimating—and the resulting velocity—is a local and independent concern. In SAFe [US: Scaled Agile Framework], however, story points must share the same starting baseline so that estimates for features or epics that require the support of many teams can be understood.” 8 Scaled Agile Framework requires normalization for sizing of the stories. if you don’t use any normalization approach for sizing of the User Stories or having different Agile methodologies for different teams (Scrum, Kanban etc), then this metric is a vanity metrics for you, and don’t add this into the dashboard. If you are following a normalization approach for sizing, then you need to measure and improve normalization of the sizing via this metric.
Committed vs Delivered: These are burn-up or burn-down charts. It shows maturity of team size estimation and helps to expose other related issues such as Blocked Items to prevent the team meeting their commitment.
Blocked Items over Time: It’s a good metric to expose the challenges around architectural runway and operations. It’s also expected that Blocked Items should be reduced by improving the Architectural Runway and empowering Agile teams.
Blocked Reasons: It’s for the investigating root cause of the blockers.
Quality Dashboard
The main audience of the Quality Dashboard is also Agile teams including System and Operation teams. Its graphs should be showing change over the selected time into Team of Teams and Team levels.
Please remember that one of the wrong ways of using the Quality Dashboard is using them to blame teams or even specific people who seem like they have caused an error. Lean-Agile mindset can strive in an environment where people take responsibility and learn from their mistakes. In an environment where people are so scared of being blamed for mistakes, people avoid taking risks and trying new ways, therefore, innovation cannot find a way to flourish. Of course, it doesn’t mean, teams and people shouldn’t take responsibility to improve the quality. These metrics are to address the issues and to asses required actions in order to avoid the same mistakes and improve the quality altogether.
Some metrics for Quality Dashboard:
Number of P1/P2 Bugs/Defects: This metric would play an important role to analyse the root cause of problems and improve the quality of development. However, it’s important to have a shared definition of Priority and Bug/Defect as well as how to treat them once they are reported. Because, for example, if all bugs/defects are assigned into a specific team, it’d be hard to track root cause and improve the quality. This kind of specialization between agile teams is not best practise either.
Number of Tests and Failed Test: Built-in Quality is one of the key values for Scaled Agile Framework as well as other Lean-Agile Software Development practises. And Test Automation is one of the effective ways to improve Built-in Quality. This metric is to show number of failed tests during either continuous integration or deployment processes.
Number of Support Requests: This metric should include call-centre calls, chats and support emails related to specific activity. However, the first responder for the problem might not be aware of the real root of the problem. To get a valuable insight to lead our next actions, this needs to be broken down into team’s work level.
Bugs/Defects Root/Types: This is also an important snapshot view which can be used to expose the root of the real problems.
Epic Overview
Epic Overview is the dashboard which includes financial metrics from Lean-Agile reporting perspective.
As mentioned before, one of the key struggles for Lean-Agile and Scaled Agile environments appears around PMO and IT Finance reports and activities. In a plan-project based development environment, it’s a relatively clear process. Based on whether they are projects or non-project changes (BAU for some organizations), they would follow different paths and be applied into different cost types (CAPEX, OPEX or sometimes even both). If it’s a project rather than BAU, a Business Case is needed to justify business requirement and to secure a budget for the project. Often assumption of the investment realisation and some projections of the Return on Investment are also added into Business Case. Business Case includes Cost based on the brief description of the Scope and financial realization or return on investment which is tightly bound to Deadlines. Therefore, the Project Manager in the plan-based project environment is the main role responsible to deliver the Business Case by delivering the project within the Cost (therefore planned resources), defined Scope and on planned Deadlines. Financial reports created by PMs, PMOs and IT Finance help Portfolio and Program Management as well other stakeholders to check whether the project is still healthy in terms of developing features as defined by scope and within planned budget. Those reports also help to validate Business Case at the stage boundaries whether it’s still valid and achievable. Therefore, the budget spent to date versus the percentage of the scope completed to date becomes the main financial criteria to measure the health of the project delivery. For example, if we spent 50% of the cost and just delivered 30% of the project features (or vice versa) then most probably something is going wrong.
Undoubtedly, to some extent we need scoping and budgeting in the Lean Portfolio and Program Management. However, the Lean-Agile approach requires quick adaptation to change via learning from customers and users. Therefore, Lean-Agile is not a development approach to deliver pre-defined, detailed scope. It focuses on delivering outcome, learning from it and adjusting the path and scope accordingly (Plan, Do, Check, Adjust Loop 9). The Scaled Agile Framework suggests creating a light-weight Business Case for Epic. Therefore, I’ll be using “Epic Overview” as a name of the dashboard for financial metrics. Epic is defined as “a container for a Solution development initiative large enough to require analysis, the definition of a Minimum Viable Product (MVP), and financial approval before implementation. Implementation occurs over multiple Program Increments (PIs) and follows [US: Eric Ries’] the Lean startup ‘build-measure-learn’ cycle.” 10
However, metrics/graphs in the Epic Overview Dashboard should include the breakdown levels into any product development or project activity which has got its own dedicated budget against a broader range of sizing and scoping. From this perspective, it can be an Epic with a light-weight Business Case or even a project with more defined scope and deadlines such as a Regularity Change Projects.
Another difference between Epic Overview Dashboard and the other dashboards is Epic Overview would show a snapshot table view rather than a graph showing change over the selected time.
Some metrics for Epic Overview Dashboard:
Initial Estimation for Story Points vs Net Story Points: These are the metrics that provide comparison between Initial Estimation of the Story Point and Net Story Points (Actual Story Points (done so far)+ Story Points at the backlog (waiting for development)).
When the Product Manager or Team is sharing initial sizing for budgeting purposes, they do not have details. Therefore, their initial sizing is just based on their experience and knowledge about the scope until that point.
Lean-Agile Product development is a dynamic process. In addition to lack information about the details during the budgeting/business case preparation stage, some features or stories might not be part of the scope anymore based on our learnings or some new ones might need to be added. This metric would help Product Managers/Owners to prioritize their backlogs.
Story Points Actual (done so far) vs Story Points at the backlog (waiting for development): This metric would be also helpful for descoping or prioritization of the backlog items.
Estimated Budget vs Actual: This metrics would be useful with other two metrics above. For example, if we already used 70% our budget but delivered half of the Net Story Points, then, we need to prioritize backlog accordingly and/or asking for additional budget.
1. Accelerate; Nicole Forsgren, PhD Jez Humble, and Gene Kim; IT Revolution, 2018; page: 12 ↩2. Lean Analytics; Alistair Croll, Benjamin Yoskovitz; O’Reilly, 2013; page: 153 ↩
3. I use Start-up Product to address a new product development in an extreme uncertainty. It can be a product developed by a start-up or a new product developed by an enterprise and aimed to be disruptive. They both have similar stages. ↩
4. “The reality is that every startup [US: product] goes through stages, beginning with problem discovery, then building something, then finding out if what was built is good enough, then spreading the word and collecting money. These stages -Empathy, Stickiness, Virality, Revenue, and Scale- closely mirror what other Lean Startup advocates advise.” Lean Analytics; Alistair Croll, Benjamin Yoskovitz; O’Reilly, 2013; page: 153 ↩
5. Eric Ries; Lean Analytics; Alistair Croll, Benjamin Yoskovitz; O’Reilly, 2013; Foreword, page: XVIII ↩
6. “For a sense of what’s possible at scale, in May of 2011, Amazon achieved a mean time between deployments to production systems of 11.6 seconds, with up to 1,079 such deployments in a single hour, aggregated across thousants of services that comprise Amazon’s platform.”, Lean Enterprise; Jez Humble, Joanne Molesky, and Barry O’Reilly; O’Reilly, 2015; page: 155 ↩
7. © Scaled Agile, Inc. https://www.scaledagileframework.com/release-on-demand/ ↩
8. © Scaled Agile, Inc. https://www.scaledagileframework.com/story/ ↩
9. © Scaled Agile, Inc. https://www.scaledagileframework.com/program-increment/ ↩
10. © Scaled Agile, Inc. https://www.scaledagileframework.com/epic/ ↩