Today, the number of competitors all building software for the same end user is greater than ever. So it’s critical that companies ensure they’re not wasting precious time and money working on features that aren’t important to their ideal customers. For this reason, many companies are adopting user-facing feedback forums. These are typically characterized by a list of feature ideas submitted by users, which are sorted by popularity based on the number of reddit style ‘up’ votes received to determine which features are most wanted by users.

On the surface, this passes our common sense sniff test; the feature that will positively impact the most users is probably the feature we should prioritize. However, as well-intentioned as this sort of prioritization method seems, it is fundamentally flawed for one critical reason: the fallacy of experiential utilitarianism. 

What is Experiential Utilitarianism? 

“What’s that you say? Extradental Youtilityan…nism?” 

I know; I hadn’t heard of it either prior to joining Parlor, but it has since become important to my understanding of work prioritization as a Product Manager. In ordinary terms, utilitarianism (you may remember it from that Philosophy 101 course you took to avoid Friday morning classes) is a theory of ethics which states that the right choice is the one that will produce the greatest good for the greatest number of people. ‘Experiential’ utilitarianism, then, simply grounds the idea of utilitarianism within the context of a user experience. That is, the thing we should do is the thing which will positively impact the experience of the greatest number of users. 

There’s just one major problem: not all users’ opinions are worth the same to your team, and surface-level reddit-style voting fails to incorporate the reality that some users are more important, knowledgeable, or engaged than others.

Choosing between what’s popular and what’s impactful

Reddit-style voting – whether within a feedback forum or not – mistakenly assumes three things about your customers:

  • They all have the same level of expertise in your product
  • They all receive the same value from using your product
  • They all bring the same value to your company

In an ideal world in which all three were true, then prioritization based on popularity (i.e. which idea receives the most votes) is certainly a viable approach. Unfortunately, it is rarely if ever the case that every user is equal across these three categories.

The question then is not what’s best for the most users, but rather what’s best for your most valuable users.

If your 5 highest paying customers request the same feature to be built and 500 users who pay you nothing request a different feature, which should you build? In this case, you’re going to build what your highest paying customers want, unless the item your free users want will significantly affect their likelihood to convert to a paying tier.

Reddit-style popularity voting is effectively meaningless if you don’t have insight into who wants what and for whom you are making decisions.

A Better Way to Prioritize

There’s nothing wrong with having all users give simple inputs on potential feature ideas. It can be extremely valuable (in fact, we do it with every potential Parlor feature). However, you want to ensure that the inputs you are collecting are meaningfully distinct, as well as tracking from whom these inputs are coming. 

Although more inputs don’t necessarily improve the clarity of the insights gained, we do recommend more than a two-tiered input system, such as thumbs up/down.

The problem with relying on simple up/down voting to prioritize your team’s efforts is that each of the inputs is forced to cover too broad of a spectrum of response.

Think about it; in a thumbs up/down system, how would you rate the following:

  • A stubbed toe
  • Being fired
  • Hitler

Hopefully you’d respond thumbs down to all of these, but in the context of the last one, shouldn’t a stubbed toe feel considerably less negative? No in-between sentiment is possible when only an affirmative and negative input is provided. That makes understanding these inputs a bit tricker than we’d prefer, and it can be easily fixed by simply adding a third (or more) input options.

How to get started

To receive meaningful and actionable feedback on the ideas you and your team are thinking of working on, here are a few simple ways to improve your early feature validation efforts:

1. Use a more diverse input method: We recommend Positive/Neutral/Negative over thumbs up/down. We’ve seen some teams adopt five or even seven point input systems, but we haven’t seen any indication this is more effective than a straightforward three point system.

 2. Limit voting to a handful of ideas you would actually consider working on right now: Having a giant list of unmoderated ideas for users to sort through and vote on isn’t practical. For example, HubSpot’s feedback forum has over 1,000 pages of ideas that have been submitted. No user is reviewing all of these. Nor is HubSpot’s team. Only ask for feedback on ideas that you would actually consider working on.

Would you look through 1,210 pages of ideas submitted by your users?

3. Identify, segment, and track your users’ inputs across different user cohorts: It’s critical to know which users are voting a certain way about potential features. Knowing how your highest paying customers or your power users feel about a potential idea should be a source to lean on when deciding what to prioritize.

With this framework in place, you’ll get more value from your users in the roadmap validation process, separating the signal from the noise by focusing on user inputs that are most valuable to your company at this point in time.