Creating a Data-Driven Organization

Chapter 6. Metric Design

If you don’t know where you’re going, you’ll probably end up somewhere else.

Yogi Berra

Count what is countable, measure what is measurable, and what is not measurable, make measurable.

Galileo Galilei

A data-driven organization needs to set out a clear strategy, that is, a direction that the business is heading, and then determine a set of top-level metrics—key performance indictors (KPIs)—to track whether the business is indeed heading in the right direction and to monitor progress and success. Responsibility for driving those top-level KPIs flows down to divisions or business units where they may define additional KPIs specific for that business unit. Ultimately, this ends up a set of operational and diagnostic metrics that monitor tasks, programs, tests, and projects that drives the KPIs.

Given this, it is imperative that metrics are designed well. They should act as true, accurate compasses. You don’t want to be following a strategic metric that indicates you are heading in the desired southeast direction, when your true heading is actually northeast, or an operational metric that indicates mobile conversion is increasing at 5% per annum when it is, in fact, flat. You don’t want to be watching a faulty diagnostic metric that fails to inform you as early as it could that your website is on the verge of crashing. Metrics are also the outputs from experiments and A/B tests which, if instrumented well, will inform causal analysis and that, as we discussed ...