You are neither right nor wrong because the crowd disagrees with you. You are right because your data and reasoning are right. Warren Buffett
Recently one of our customer asked for help in building a business case for their ALM transformation project. The customer wanted to prove to their stakeholders that the required investment in ALM was worth it.
We started by providing some “out of the box” statistics but we also asked them why and what they wanted to achieve, bringing their own context into the equation.
At first the customer thought that using TFS to gather all the data, and turn it into information, would suffice. However, going through the process it became clear it would not. The end result was far from what they originally imagined, however it was aligned with what their goal was, i.e. prove the value of investing in ALM being done right. So what did we do to get there?
Before we start, remember that all models are wrong but some are useful (George E. P. Box), there is no cookie-cutter approach to measurement nor there is a silver-bullet to solve challenges.
Why the GQM approach?
We decided to use the Goal Question Metric (GQM) method/approach to help the customer define its goals and metrics. GQM is an approach to software metrics developped by Basili and Weiss (1984), then Solingen and Berghout (1999) that defines a measurement model on three levels:
- Conceptual level (goal): A goal is defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view and relative to a particular environment.
- Operational level (question): A set of questions is used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.
- Quantitative level (metric): A set of metrics, based on the models, is associated with every question in order to answer it in a measurable way.
Other approachs like the Value Focused Thinking (VFT - Keeney, 1992, Keeney, 1996) approach and The project objectives measurement model (POMM - Barclay, C. and Osei-Bryson, K-M 2008), could have been used. Even if GQM is intended for software organizations, the focus was more project/process management than true software measurement (e.g. lines of code) and we had experience with it.
Step 1 - Definition of the Goals, Questions, Metrics
First, we started by defining the goals, why they wanted to improve, why the investment was needed. Second, based on those goals we defined the questions they wanted to answer, what needed to be looked at. Third, we defined the metrics they needed to measure to help answer the questions and ultimately meet their goals. For example:
Goal:
- Data/factors: Cycle time is poor, customers are waiting too long, people complain about wait time. Defects resolution is more important than new feature right now.
- Goal 1: Reduce cycle time to resolve defects
Questions:
- What is the average cycle time to triage a defect? To resolve it? To close it?
- Is there a part of the application where defects are clustering?
Metrics:
- Average days from when it is reported to when it is triaged
- Average days spent in active state
- Average days spent in resolved state
- Number of defects by application module
- Number of defects by root cause
Keeping in mind that the software development methodology and process had an impact, i.e. the metrics could be different for an Agile project vs. a waterfall project.
And ended up with something like this:
Step 2 - Metrics prioritization and data research
We were not done though. We realized that we had many metrics that needed to be prioritized, so we looked at low hanging fruits and importance/impact on the business, weighting them to get to priorities. We then looked at the “balance”, for example:
- Leading vs. lagging vs. current – making sure we were taking time and process steps into consideration.
- Governance vs. management vs. process vs. technical vs. operations vs. … - making sure we addressed all the necessary functions of the software organizations.
- We then looked at how to get the data and from where/which system, it became clear that TFS was not the only answer.
- We re-prioritized again based on the how, being careful not to only include what was measurable (easy to do) but more importantly what was meaningful.
- We looked at how to report the data, what had to be represented in graph, what kind of graph, timeline and intervals to collect data, showing trends or detailed values, consolidating on a dashboard, etc.
- We left out targets for the first phase, we first needed to collect historical information before we could set meaningful targets, regardless if we decide to do it per team, per process, per applications, etc. and how we decide to do it (thresholds, peer-pressure, etc.).
- We also defined the implementation plan: change in process, training required, systems to be customized, etc.
Takeaways
- We decided to use GQM but don’t be married to a model.
- Like my former esteemed colleague used to say “it’s like apple and tractors, they are both red/green but they don’t do the same thing”, make sure to compare apples to apples, for example if you measure “bugs” how you manage them can be different in Agile and in Waterfall.
- Tools are not silver bullets, don’t confuse measurable with meaningful.
- GQM is easy to understand but not that easy to put in practice, you can get off-track easily, jumping to metrics directly without defining and challenging the goals. Make sure to always ask yourself why and get a third-party to assist so you get a pair of fresh eyes.
- You can download this spreadsheet that provides GQM examples and a list of actions: Goals-Questions-Metrics example
Finally, “You got what you measure, measure the wrong thing and you get the wrong behaviors.”
― John H. Lingle
