Part 8
Measurement
Principles of honest measurement
- Baseline first, collected before the intervention is announced, because announcement is itself an intervention.
- Predict, then measure. Write down expected direction and magnitude in advance. Retrospective interpretation always finds success.
- Leading and lagging. Lagging indicators confirm; leading indicators steer. Only the leading ones let you act in time.
- Balancing measures. What could this make worse? Every improvement can export harm; measure the export.
- Plot over time, not before-and-after. Two bars cannot distinguish a real shift from normal variation.
- Aggregate, do not surveil. Team level and above. Individual-level OD metrics change behavior toward the metric and destroy the data.
What to measure
| Category | Examples | Notes |
|---|---|---|
| Perception | Engagement, psychological safety, Westrum, eNPS | Cheap, fast, gameable |
| Behavior | Reporting rate, speaking-up incidents, review latency | Harder to collect, much harder to fake |
| Flow | Lead time, cycle time, queue length, work in progress | Objective; needs instrumentation |
| Outcome | Harm events, defects, customer satisfaction, retention | What matters; slow, noisy, multi-causal |
| Capability | Cross-skilling, bus factor, time to onboard | Predicts resilience |
| Cost | Turnover cost, agency spend, rework, overtime | Speaks to finance; easy to overclaim |
Behavior measures are the most under-used and the most informative. "Did anyone challenge a senior decision this month, and what happened to them?" is worth ten survey items.
Goodhart's law and metric distortion
"When a measure becomes a target, it ceases to be a good measure."
Predictable distortions: gaming (hitting the number without the outcome — four-hour emergency targets met by moving patients to a corridor reclassified as a ward); tunnel vision (measured things improve, unmeasured things decay); effort substitution; and suppression (reporting falls because reporting looks bad).
Countermeasures: balanced sets rather than single numbers; pair every efficiency measure with a quality measure; never tie individual pay to a single OD metric; and work out in advance what the metric would look like if gamed, then watch for that signature.
The reporting-rate paradox
The most important measurement trap in OD.
When safety culture improves, incident and near-miss reporting rises, because people are willing. When it deteriorates, reporting falls.
A rising reporting rate is therefore success and looks like failure.
Agree the interpretation with the board before the data moves. Pair reporting rate with a severity-weighted harm measure: reports up, serious harm flat or down, is the signature of a healthy system.
This applies beyond safety. Rising numbers of raised risks, escalated concerns, disclosed project problems, and reported near-misses in any domain are usually evidence that the culture work is functioning.
Domain metric sets
Health care. Pair harm outcomes (lagging) with reporting culture and staffing fill rates (leading). Use statistical process control — already standard in clinical improvement — rather than month-on-month comparison. Always include a balancing measure; throughput improvements frequently export harm to readmission or to another department.
Software: DORA metrics. Throughput and instability, which the research found move together rather than in tension:
- Deployment frequency — throughput
- Change lead time — commit to production; throughput
- Failed deployment recovery time — throughput; earlier called time to restore service, and before that MTTR
- Change fail rate — instability
- Rework rate — unplanned deployments in response to an incident; instability, and the most recent addition
Plus reliability — availability and performance against user expectations.
Recent reports add a caveat worth carrying into any AI rollout: AI-assisted development correlates with higher throughput and higher instability, because code arrives faster than review and deployment capacity can absorb it. Read a throughput jump against the two instability metrics before calling it a win.
Why this matters to OD: the research established that Westrum generative culture predicts delivery performance, and delivery performance predicts organizational performance. This is one of very few places in the OD literature with a quantified chain from a culture construct to a business outcome on a large sample. It also found that formal change approval boards were negatively associated with performance — an uncomfortable and well-replicated result.
Use DORA at team level, as trend rather than target, all four together, derived from the pipeline rather than surveyed where possible. Never rank teams on a dashboard: different teams have genuinely different constraints.
Knowledge work generally: the SPACE framework. Five dimensions — Satisfaction and wellbeing, Performance, Activity, Communication and collaboration, Efficiency and flow. The rule: pick metrics from at least three dimensions, and include at least one perceptual measure. No single number describes productivity.
Satisfaction is listed first deliberately: a productivity gain purchased by burning people is a loss on a delay. Activity is the easiest to collect and the most misleading in isolation.
SPACE generalizes directly beyond software. Health systems measure activity extensively and flow-interruption almost never, which is one reason clinician frustration is invisible in management data.
Executive. Demand three things of any people program: the baseline, the prediction, and the balancing measure. Programs that cannot supply all three have not been designed, only purchased. When asked for a single productivity number, offer three to five spanning dimensions instead, and explain that the single number will be optimized against.
See topics/dora-metrics, topics/space-framework, and topics/measurement-and-metrics.
Evaluating an OD program honestly
Kirkpatrick's four levels, applied to OD:
- Reaction — did people find it useful? Nearly meaningless alone, and the only level most programs report.
- Learning — did knowledge or attitudes change?
- Behavior — is anyone doing anything differently, 90 days later? The level at which most programs are shown to have failed, which is why it is rarely measured.
- Results — did the organizational outcome change?
Add a fifth question Kirkpatrick does not ask: would it have changed anyway?
Options for causal confidence, ascending:
- Pre/post in one unit — weakest, confounded by everything.
- Pre/post with a comparison unit — much better, usually feasible.
- Staggered rollout, units acting as their own controls — strong and practical, since rollouts are staggered anyway. Stagger deliberately and you get a natural experiment for free.
- Randomized allocation of units — strongest, occasionally possible, and worth arguing for when the intervention is scarce and must be rationed regardless.
On engagement surveys specifically. Trend beats absolute; variance beats mean, because the organizational average hides that team A is at 8.5 and team B is at 4.2, and the variance is the actionable finding; the local manager effect dominates enterprise policy; response rate is itself data, and a falling one usually means the last round produced nothing. Report the action completion rate alongside the scores at board level — it changes behavior more than the scores do.