Measure What Matters got 2 things wrong

Rather watch a video? Click here!

Measure what matters is in many ways a great book. I consider it a must-read for anyone new to OKR. Unfortunately, there are a couple of things that Measure What Matters got entirely wrong.

Problem 1: Confusing outcomes with outputs

First of all, John Doerr confuses outcomes with outputs. A lot of the examples presented in the book contain Key Results that measure outputs. An output is something you do, an outcome is something that happens as a consequence of what you do. It’s very tempting to have outputs as Key Results, but the problem is that completing them does not guarantee they’ve created value for your business..

For example: “I’ve had a call with a prospect” is an output, “I’ve closed a customer” is an outcome. Doing calls with prospects probably helps you close customers, but would it help the company if you’ve had 250 calls and closed no deals?

The problem with measuring outputs as Key Results is that they’re not measuring the things that matter to the company (the title of Doerr’s book, therefore, is a bit unfortunate).

Another problem with measuring outputs as Key Results is that they lock you into the how. If you’ve had 250 calls with prospects but closed 0 customers, you should try something else to see if other approaches result in more customers closed.

Introducing Initiatives

At Perdoo, we call outputs Initiatives. An Initiative can be a task, project, deliverable, or any sort of activity that you execute to help you achieve your OKR. We believe it’s important to track your OKRs and their Initiatives in the same place: it enables you to iterate what you’re doing until you see your Key Results move.

Let’s have a look at an example from Measure what matters. On page 227 of his book, you’ll find the following OKR:

OBJECTIVE

‍Institute a culture that attracts and retains A players

‍KEY RESULT 1

Focus on hiring A player managers/leaders.

KEY RESULT 2

Optimize recruitment function to attract A-player talent.

KEY RESULT 3

Scrub all job descriptions.

KEY RESULT 4

Retrain everyone engaged in the interviewing process.

KEY RESULT 5

Ensure ongoing mentoring/coaching opportunities.

KEY RESULT 6

Create a culture of learning for the development of new and existing employees.

Literally, ALL the Key Results here are outputs (or Initiatives). Some of them, like Optimize recruitment function to attract A-player talent could even be Objectives of their own. What if all these 6 Key Results would have been a 100% completed? Would that mean you’ll have a culture that attracts and retains A players? Of course not.

Sure, there is an implied relationship between the Objective and these 6 Key Results. I can imagine that by completing these Initiatives you think your culture is better at attracting and retaining A players. However, to know if that is the case, you’ll need to look at outcome-based Key Results that measure, for instance, if the job acceptance rate for A players has improved. If you’re putting in all this work, wouldn’t you want to know whether it actually helped you achieve the Objective?

You’ll find similar problems with the examples provided on page 108, 109, 111, 121, 132, 205, 218. Admittedly though, not all the examples are bad — you’ll find a good OKR on page 210.

OBJECTIVE

‍Delight customers

‍KEY RESULT 1

Net Promoter Score of 42 or better.

KEY RESULT 2

Order Rating of 4.6/5.0 or better

KEY RESULT 3

75% of customers prefer Zume to a competitor in blind taste test.

At Perdoo, we’ve developed a simple process that ensures that your Key Results are outcome-based. It also makes sure that you always start with the end in mind. Simply ask yourself the following questions (in this order):

1. Where do I want to go? The answer is your Objective
2. What are the results that I need to achieve in order to get there? Key Results
3. What do I need to do to achieve those results? Initiatives

If you’d like to be a company that is results- instead of activity-driven, you should not confuse outcomes with outputs.

Problem 2: Key Results turning into Objectives

The second problem I uncovered is that John Doerr makes Key Results turn into Objectives if they are assigned to someone else (or if an Objective is aligned to another Objective).

Whether something is an Objective or a Key Result should not depend on where it appears in a hierarchy. An Objective is an Objective, because it meets certain criteria. Creating OKRs is already hard, you’ll leave the organization completely confused as to whether an Objective is an Objective or a Key Result, depending on whose point of view you take.

Quantitative Objectives

Key Results should always be measurable. Turning Key Results into Objectives automatically results in quantitative Objectives. This creates all sorts of other problems.

Key Results must always contain a metric and a target (as far as I’m aware, Doerr agrees). So if a Key Result can become an Objective, that Objective will also contain a metric and a target. The person or team who will own that peculiar Key Result/Objective, will create another set of Key Results for it. So you’ll end up with this structure:

- Objective A
--- Key Result A1
--- Key Result A2 / Objective B
----- Key Result B1
----- Key Result B2
----- Key Result B3

Doerr explains on page 120 that progress for the Objective is based on the progress of an Objective’s Key Results. No matter what metric you put in Key Result A2/Objective B, Key Results Bto B3 will have to contain the same metric in order to accurately measure progress for Objective B. For obvious reasons, this almost never works.

In this article, I explain in more detail why Key Results cannot become Objectives for someone else.

Relevancy

When you want a Key Result for person A to turn into an Objective for person B, it typically means you are setting OKRs on different levels in your organization (e.g. a CEO’s Key Result becomes an Objective for someone reporting to the CEO).

First of all, I know Doerr prefers to talk about a “CEO’s Objective”, but I prefer to call it a Company Objective. OKR helps you unite your organization behind your common Objectives, which is not what a CEO Objective symbolizes. I’ve also learned that individual employees are more motivated to help their company achieve its OKRs, instead of helping their CEO achieve his or her OKRs.

Second, if you set an Objective on company-level, you should do so because you consider it’s a priority for the entire organization, and you want several teams to focus on it. When you assign a company-level Key Result to a team, you’re basically saying that only that team is expected to contribute to that Key Result. Why is it then a company-level Key Result? Shouldn’t it be a team-level Key Result? Company-level OKRs should be relevant for the entire organization, or at least the majority of teams and employees.

The purpose of the Objective

Last but not least, there is a reason why real, qualitative Objectives exist. They have a quality of their own: provide direction and keep everyone focused on the bigger picture. Quantitative Objectives (i.e. Objectives that are actually Key Results) are not able to serve this purpose, as they are metrics-based and contain a clear target.

At Perdoo, our Coaching team had an Objective in Q3 2017 to Create an awesome onboarding experience for new customers. The Key Results were to have (i) Avg. NPS after 3 months at 50, and (ii) Avg. Onboarding rating at 8 out of 10.  Imagine we did not have the Objective. All the Coaching team would have focused on was to make sure that NPS would be at 50 after 3 months, and to receive an Onboarding rating of at least 8.  They would have had no clue why these Key Results mattered. How can you keep your entire organization on the same page if people are not able to see the bigger picture? How can you stay aligned if people are not given any sense of direction? How do keep your team motivated if don’t tell them why certain results matter?

Conclusion

Measure what matters offers a great introduction to OKR, and thoroughly explains several major benefits that organizations can derive from adopting the methodology. However, when it comes to how OKR should be applied across an organization, Doerr’s approach has some serious flaws.

One of the major advantages of starting to work with OKR, is that you cut out the BS and start looking at what’s really happening. You move from an output-oriented towards an outcome-oriented organization.

Another major advantage is that OKR provides direction to employees on different levels, keeping teams across different departments and geographies aligned. It enables everyone to see the bigger picture and helps them make impactful contributions to their company.

It’s hard, if not impossible, to achieve that with Doerr’s approach.

Watch video