Without big data, you are blind and deaf and in the middle of a freeway. - Geoffrey Moore
Many would agree with Moore’s position on the importance of data, evidenced by the trust placed in the breadth, depth, and diversity of information collected by most organizations.
The trouble is that all data is subject to interpretation; and without considering WHAT, HOW, and WHY we measure, the information gathered quickly transitions from promising to perilous.
Four typical mistakes tend to undermine our choices about WHAT, HOW, and WHY to measure:
We don’t link our measures to our strategies
We do not validate the causal links between measures
We start collecting data before knowing what we want to find out
We measure incorrectly (i.e., we don’t use the right methods for the circumstances)
While the implications of these mistakes range from inconvenient to disastrous, in all cases the result is a false impression of the issue of interest.
We are missing part of the picture; and we can’t fix what we don’t see.
Fortunately, there are seven steps that can help us overcome the measurement mistakes built into many organizational systems.
1. Get clear on WHY you are measuring
What are the big questions you are trying to answer?
What need(s) are fueling these questions? (self-check: are these the right questions for those needs?)
How will the information be used, once collected?
Who will use this information? (self-check: what are the different ways different stakeholders will use the information?)
2. Create a causal framework
A causal framework outlines the relationships between the variables believed to impact your target outcome(s):
Clarifies how the actions we take should impact the target outcome(s)
Encourages us to test our assumptions about relationships between variables
Flags covariates that might interfere with the desired outcomes
Identifies potential biases (errors) in the system
Exposes any gaps in the causal chain (i.e., measures we did not consider)
3. Pull together the data
Start with what you have by taking an inventory of the measures already being tracked within the (e.g., performance measurement, information - purchasing, manufacturing, customer service, etc.)
Tip: Do not limit yourself to the data being tracked by the target department (e.g., HR, Learning & Development). Often, there is information gathered for other purposes that can be leveraged.
4. Turn data into insights
Apply statistical analyses (e.g., regression, correlation) to expose patterns and test relationships
Tip: Be careful about the interpretation of “significance”. Statistical significance (or insignificance) does not always equate to real-life significance. Context matters!
Use qualitative analyses (e.g., focus groups, 1:1 interviews) to test hunches, explore patterns, and add depth to the data; and remember that insight is context dependent.
What else is going on?
Why does that matter?
What do these results mean right now, with this population, in this environment?
5. Base actions on findings
Take actions based on the findings and identify the specific measures (already being tracked) to use for feedback
Tip: Focus on 1-2 actions at a time, to avoid misappropriating any observed effects.
6. Assess outcomes
Evaluate the outcomes of the actions taken and make adjustments as required
Tip: Use the causal framework to explore all variables related to the actions taken. Not seeing the desired outcomes does not always mean the actions were wrong (e.g., other factors might be interfering with an otherwise strong set of actions).
7. Refine the causal framework
A causal framework is not set in stone. New information can change the framework (e.g., changes to the competitive environment can strengthen, weaken, or neutralize the effectiveness of previously key activities)
Tip: Performance measures might also need to be refined to support an adjusted framework (e.g., to make sure performance is being tracked against drivers of key activities)
Just starting to measure?
These seven steps can be applied to all types of organizational measurement (e.g., performance assessment, organizational health, culture, change, leadership), regardless of scope.
Start small by choosing an area believed to have a strong, positive affect on the organization’s performance (e.g., employee satisfaction)
Identify a series of small activities believed to improve it (e.g., training)
Measure the effects of those activities