We’ve all heard the old saying, you can’t improve what you can’t measure. That’s particularly fitting for us as product managers. Arguably one of the most important aspects of our job is measuring the general, long term success and improvement of our products. As such, there’s been a rise in popularity of a few core product engagement metrics that many teams adopt. Whether it’s NPS to track customer satisfaction or some temporal engagement metric like daily/monthly active users, these metrics have become ubiquitous due to a product team’s need to understand whether what they’re building is actually making an impact.

However, while these traditional metrics are meaningful for many product teams, for others, they simply don’t make sense due to the unique nature of their products or user populations.

What then do we do when traditional product metrics don’t align well with our products?

Rather than forcing these traditional metrics to fit (like forcing a square pin into a round hole), teams would be better off constructing their own engagement metrics which better align with their unique product circumstance.

In this post, we’ll explore an alternative to the more traditional product metrics: outcome driven product metrics.

Why traditional product metrics don’t fit 

Many companies focus on frequency or consistency based product engagement metrics, such as:

  • Daily/monthly active users 
  • Average session length 
  • Number of logins over a specific period of time

The primary reason these metrics have become popular is because, in general, the number one indicator of anti-churn is an increase in the frequency or consistency in product engagement. For most companies, a more active and deeply engaged user population leads to higher renewal rates, referrals, and upgrades. For many companies, the value of these metrics goes even further: an increase in any of these metrics directly relates to an increase in revenue due to the nature of their business models. Let’s use Instagram as an example.

Instagram’s business model relies on paid advertising, whether that’s cost per impression (CPM) or cost per click (CPC). In Instagram’s case, focusing on improving metrics like daily active users or average session length drives corresponding increases in these advertising metrics, leading to more revenue. By improving on these metrics, the outcome is more users spending more time on Instagram, seeing more paid ads, and clicking on those ads. All of which increase the bottom line of Instagram. 

However, not every business model aligns well to metrics that measure daily active users or average session length.

For a large number of products, these more frequency or consistency based product metrics don’t align well with the type of product they have, typical user engagement they see, or the uniqueness of their user population in the sense of some demographical or behavioral sense.

Let’s look at an alternative example in which traditional metrics don’t work for the product in question – TurboTax

TurboTax is designed to make filing your taxes as easy and convenient as possible without having to talk to a tax professional. It is widely considered and has been so for years as a best in class solution for its industry.

However, because tax season comes once a year, traditional product metrics like daily/monthly active users don’t make sense. Why then would the product team at TurboTax hold itself accountable to it?

In the case of TurboTax and companies like it, which have a highly seasonal engagement pattern, metrics that measure daily/monthly active users don’t align well with both the outcome users expect to achieve while using the product or TurboTax’s business model. While the more traditional product metrics may shed light on how TurboTax usage is increasing within a very narrow window of time, they don’t accurately measure if the product is succeeding in ensuring the process of filing taxes was less difficult.

For many products, not just seasonal ones like TurboTax, we recommend using a different type of product metric – outcome driven product metrics.

Outcome driven product metrics

Every product is built with a primary goal in mind, typically in the form of trying to solve a functional or emotional need for its end user. Outcome driven product metrics measure the success of the reason why your product was built, that is they measure whether your product is successful in achieving the need it was built to solve. 

We can start to understand the value and importance of outcome driven product metrics by looking at a successful Boston startup, Firecracker, as an example.

The Importance of Outcome Driven Metrics in Atypical Products 

Parlor’s co-founders previously worked for a Boston based EdTech startup called Firecracker, which built an adaptive learning platform to help medical students prepare for their licensing board exams in the most time-efficient way possible. In order to pass these board exams (taken once every 2 years), students needed to master a massive number of medical topics. 

Firecracker’s algorithm ensured students were only assigned the most important topics to study each day – either because they focused on a student’s weak areas or an area which they hadn’t reviewed in awhile. This satisfied students’ biggest need – minimizing the amount of studying time required to get as high of a score as possible on the exam. 

Why traditional metrics didn’t work at Firecracker

It may not be immediately obvious why, but traditional product metrics did not apply well to Firecracker.

While many teams would view an increased amount of time spent in product (either in number of logins or average session length) as a sign of success, Firecracker’s product team sought to minimize the time users spent in the product.

The better the algorithm, the better the study recommendations, meaning the less time users need to spend in product studying. However, decreased product engagement was only a sign of success if student outcomes on the exam (i.e. their final score) stayed consistent or improved as study time decreased.

The success of the Firecracker product, then, could only truly be measured every two years when a correlation between average study time and board exam scores could be determined. A successful outcome would not only prove the value of the product, but also lead to a massive increase in product referrals (roughly 70% of all Firecracker customers came in through direct referral from other students). 

For these reasons, Firecracker’s team stopped tracking traditional engagement metrics focused on increasing the amount of time users spent in product. Instead, they focused on measuring the relationship between the amount of time students spent studying with Firecracker and the resulting board score. This outcome driven metric was much better correlated to the overall success of the product they were creating, and also better aligned with the number one driver of revenue, which was student referrals.

The North Star of product metrics

Every team should have at least one outcome driven product metric even if your team still uses the more traditional ones.

To get started, create a metric that directly aligns with or results in your users’ ideal outcomes, whether those be functional or emotional.

If you focus on why your product exists, you’ll be guided to the metric that measures why you’re being successful.

The outcome driven metric can act as your product’s North Star. Whenever you’re conflicted about what to do next to your product, for example deciding what feature in your backlog to build next, use your outcome-driven product metric as the de facto decision maker to help break the log jam.

This North Star metric will have the highest correlation to long term user satisfaction and customer loyalty.

Outcome driven product metrics measure what matters most – whether your users are continuously reaching their ideal outcome by using your product.