The value of Output vs Outcome when choosing KPIs

Output is underrated these days. In most product-organization or modern-leadership-related literature the gist all too often seems to be something along the lines of: stupid/old organisations produce output, smart/modern organisations produce outcome. Take, for example, Marty Cagan (who is a well-known author and a founder of the Silicon Valley Project Group), where he compares (old) project teams and (modern) product teams, summarizing: In contrast, with the product model, we focus on products and outcomes. 

I think I know where this spin comes from and it does have merit– I’ll get to that in a second. But in my observations, it has an overshooting effect on product organizations to view output as an inferior or even outdated type of goal. I think this is a dangerous misconception. Output matters just as much as outcome. Particularly for a startup.

But first let’s clarify what is what: Output is a (quantifiable) result of your efforts. Outcome is its value for your customers. Output is a cherry cheesecake you’ve been baking for 2 hours. Outcome is a cherry cheesecake that your family loves. The key difference: the arbiter of output is yourself – you can measure it internally. The arbiter of outcome is the customer – you will need external input to measure success.

From this simple distinction, it is easy to see why outcome is so cherished by product-organization evangelists: focusing on outcome makes an organisation move away from a stakeholder (internal) to a customer-centric (external) model, and it puts the solution-finding into the hands of the teams (and away from management). That in itself is not a bad thing. Also, it just makes sense, doesn’t it? After all, why would you focus on baking a cherry cheesecake that no one likes? Ultimately, it’s the result that counts, and for a business, this result is that the customer likes what you’ve cooked up. So much that enough customers are willing to pay enough money for it for you to reach your strategic goals.

But here’s the thing: how do you get there? How do you get to the point where you are building something that your customers love? You can’t make them love it. It’s their decision. And you won’t know in advance what exactly it is that motivates this decision because customers usually only know what they like if you’ve already built it for them and they’ve tried it.

The answer is of course: you must find out by testing and adapting your approach (following a strategy). If you are a startup you must do so successfully before you run out of runway (money that you can spend until you break even or have achieved some of your goals in a way that will convince investors to buy shares of your company at a high valuation). And that’s why output matters: your chances of success are bigger if you can produce more tests with the same amount of money and time. If you cab get more output from the same input.

Outcome is what you pursue (and you can’t lose track of this). But output covers and measures the process of getting there. And because nothing about creating great products or services is easy and details matter, you cannot take the eye off the process believing that the right kind of goal somehow will get you there.

What does this have to do with KPIs?

Again, it seems to me that the spotlight on outcome has motivated product organizations to seek “smart/motivating” outcome-related KPIs by default. But choosing output KPIs over outcome KPIs and vice versa should be an informed and analytical decision on a case-by-case basis. Not one of conviction.

And for such an informed and analytical decision we must be clear on the the pros and cons of output- and outcome-related KPIs:

Cheat Sheet Outcome&Output KPIs

Output-related KPIs are great for tracking progress and bad at capturing the value of your actions. They can feel repetitive and boring and they need additional guidance and measuring of success against the goals of the organization in the long run.

Outcome-related KPIs are great for focusing on value-to-customer, but they often relate poorly and with delay to the efforts made. They are therefore bad at tracking progress and potentially destructive for accountability and motivation.

You might be tempted to think that this does not apply to outcomes that you can measure by way of your traffic data - after all these consist of the behaviours of your customers and they are readily available in your analytics suite in pretty much real-time. But in reality, most results and in particular most A/B tests take a while to be statistically relevant. Also sometimes have to wait a bit to let customers get accustomed to the changes you’ve undertaken - it is not uncommon for numbers to go down after a change and to stabilize only after a couple of days. Last but not least, traffic data can be impacted a lot by effects outside of your current test setup such as changes to the channel mix and channel interdependencies (marketing campaign, visibility/SEO changes), seasonality, weather, large sports events, and competitor activity, which is particularly impactful if you are looking at changes over time instead of an A/B comparison.

If you are still unsure which KPI to use try this short decision tree:

Distinguish between short and longer-term goals.

• If you pursue a goal that you can reach in a couple of weeks a (resource-saving) output KPI should be your starting point. Only if you think that you will get a better quality of result measurement with an outcome goal (this will mostly be the case if the causality between measurable output and desired outcome is too unclear) consider switching to an outcome KPI. In that case, you must be reasonably sure that you are aware of the causality chains having an impact on this KPI and that you are capable of adjusting for these and still have stable and meaningful results. If you are doubtful whether you can achieve this stick to the output KPI despite its shortcomings - it’s better (and less frustrating) to be unsure whether you are heading in the right direction than to be unsure of whether you are creating progress at all.

• If reaching your goal requires more time or is an ongoing effort, combine output and outcome goals. Use output KPIs to track progress (immediate/recurring goal) and outcome to verify your assumption that this progress leads to the desired goal (control marker). This way the qualities and deficiencies of both types of goals complement and support each other.

For avoidance of doubt: no, this is not the same as distinguishing between Objectives and Key Results (OKRs methodology). Without getting into the topic too deeply: Objectives tend to be outcome-focused but they can also describe desired outputs (“release a new product line”) and Key Results can be either output-focused or outcome-focused, depending on what they are measuring. But you can use OKRs in the manner suggested above by making Objectives outcome-focused and using output KPIs for Key Results.

Pursuing a goal with clarity and conviction is a huge part of success. But only one part. The other is being diligent and focussed on every step of the way. And so you will need to focus on outcome AND output. If you choose between the two do so based on their strengths and weaknesses and not because one is more sexy than the other. If you can, combine the two to monitor direction as well as speed, knowing that directional tracking might take longer to show. Good hunting!