While ROI is measured from the bottom up, ROI evaluation is planned from the TOP DOWN.
Planning ROI evaluation involves mapping the impact value chain. The impact value chain is an outcomes hierarchy linking (outcome) measures to operational metrics (consequences), and on to strategic metrics (impacts).
The impact value chain is mapped in reverse-order:
Identify the strategic metrics you want to impact
Identify the operational metrics that contribute to those strategic metrics
Identify the outcome measures comprising the identified metrics
Identify the behaviours that directly address outcome measures
Identify any other factors that might affect the strategic or operational metrics, or the outcome measures
Using the identified outcome measures, operational and strategic metrics:
Identify the KPIs – the operational metrics that provide a direct line of sight into progress vs. business objective
Identify leading and lagging indicators
Identify counter-metrics
Identify relative measures
Determine whether the relative measures and/or counter-metrics should be tracked based on the extent to which they will help fill in the picture of ROI
Choosing Measures
The impact value chain should include all three levels of ROI inputs and provide a balanced view of the total impact of a program.
A balanced view involves asking “if the program has the desired effect, what else would happen?” and “if the program does not have the desired effects, what would happen?”. The answers to these questions point to other observations, measures, and metrics that can be used as check-points, to ensure the impact value chain is progressing in the right direction.
Rival theories are alternative explanations for the observed outcomes (e.g., poor behaviour change might be a process problem vs. a skill problem post-learning intervention). These should be monitored, weighed, and potentially reported on using two additional types of measures:
Relative measures show predictable patterns relative to the outcome measures, providing context and meaning for the observed outcomes (e.g., employee engagement and employee retention tend to go hand-in-hand)
Counter-metrics move in the opposite direction to the measures or metrics being used to evaluate ROI, providing real-time checkpoints to ensure the right progress is being made (e.g., as employee engagement goes up, turnover tends to go down)
Tip: ALL measures and metrics should be a combination of quantitative (e.g., cost, margins, participation) and qualitative (e.g., employee commitment or morale; resistance to or adoption of change) inputs. A strictly quantitative or purely qualitative approach will always miss half the story. The more assumptions made early on in the ROI evaluation process, the more the validity of your ROI calculations should be questioned.
Why You NEED a Measurement Strategy
Measuring ROI is no small task, so it’s important to be sure your investment in the ROI process is worth its return. Yes…the ROI must have ROI!
IMPORTANTLY, this does not mean the results of the ROI evaluation have to be good (that’s a separate issue); it means the insights must warrant the 1-2 year-long commitment that is ROI evaluation.
Not every program, organizational initiative, or learning intervention is suitable for an ROI evaluation. For example, a specialized technical training course for 10 help desk agents within a call centre team of 500 has a low likelihood of having a direct, material impact on organizational metrics – no matter how robust the learning might be. In contrast, an enterprise-wide mentoring program with a 90% uptake across the country, linked to a 2-year, 4-level leadership development curriculum is better positioned to have a more reliable impact value chain.
A measurement strategy sets the organization’s position on measurement (what should be measured, when, why, and how) and provides a framework for:
Evaluating current programs
Assessing the extent to which new programs should be measured
Creating programming that is measurable
Identifying ROI evaluation-worthy programs
Using the insights derived from different levels of measurement to optimize the utility and effectiveness of organizational programs
If you have already undertaken an ROI evaluation, you have done up to 50% of the work involved in establishing a measurement strategy.
If you are new to ROI evaluation, you have an opportunity to establish a valid framework for program evaluation without the acute pressure of reporting results.
In either case, the purpose of a measurement strategy is not to evaluate ROI; rather, the point is to have a planned, consistent approach to measuring. Sometimes this will mean measuring all the way to ROI. Most often, it will mean measuring enough to gain insights about the organization’s needs, performance, and “health”; allowing you to optimize the effectiveness of the programs offered.
**This is part of the PROVE IT! Series of posts on measuring and using ROI to generate organizational insights and drive strategic decisions.
