How to Measure Feature Success
The best first step to building a data-driven culture
Understanding the Need for Data
Heap understands the challenges product teams face when adopting data-driven decision-making. Whether you're new to data, joining a team with varying data literacy, or responding to executive requests like "Hey! I want to see the data!", this guide provides a clear path forward.
For Product Managers and digital teams, common scenarios include feeling that data would be beneficial but not knowing where to start. Amidst high-level marketing and numerous metrics, it's easy to feel overwhelmed. This guide offers an easy, nearly-foolproof way to begin.
A Low-Pressure Framework for Success
Based on extensive work with customers, Heap offers a framework designed to be the least stressful approach for individuals and team leaders. The focus is on keeping things low-pressure, starting with understanding what and how to measure, with evaluation coming later.
The goal is to adopt low-lift techniques that introduce data and metrics into your process. Once comfortable, you can use these metrics to set goals, refine features, plan roadmaps, and document impact.
This guide provides the foundation needed to become a data-driven machine. A companion worksheet is available to help track metrics and establish baseline numbers.
Key Steps to Measuring Feature Success
Step 1: Measure What You've Already Launched
While using data for future decisions is a long-term goal, a great starting point is to gather data around a feature that has already been launched. This shifts focus from "output" (shipping features) to "outcomes" (business impact).
Shipping a feature is an accomplishment, but its true value lies in whether it helped the business. Tracking metrics is the only way to know this. Basic metrics like user count, user demographics, ease of use, and user satisfaction can be highly informative.
Collecting data allows you to set baselines for measuring future feature effectiveness.
Step 2: Assess Your Feature Across Five Dimensions
A framework is recommended to measure success across five key dimensions. While not all may apply to every product or feature, teams often intuitively track some while neglecting others. It's beneficial to discuss which dimensions are most critical for your business right now.
For each dimension, identify the questions you need to answer and how to gather the necessary data. Heap provides resources and ideas for data collection.
The five dimensions are: Breadth, Frequency, Depth, Usability, and Sentiment.
Step 3: Start Sharing and Iterating
With data in hand, you can begin improving your feature. Consider issues like discoverability, cohort differences, usage frequency, and user journey completion. Use data to form hypotheses, create experiments, prioritize efforts, and report on your work.
The ultimate aim is to think about outcomes over output. Becoming data-driven is a journey, and taking the first step is commendable.
The Five Dimensions of Product Success
1. Breadth
Definition: Breadth measures the number of users who have adopted your feature or the number of users per account. It's often tracked as "adoption."
Importance: It indicates how widely a feature is adopted, if target users are using it, and can signal discoverability issues, communication gaps, or usefulness. From a business perspective, it relates to "stickiness" and churn risk.
Potential Questions:
- How many users overall have seen or tried the feature?
- How many users per organization have seen or tried it?
- What overall percentage of your users is that?
- If specific user segments were targeted, how many of them have seen or tried it?
2. Frequency
Definition: Frequency measures how often users engage with your feature within a given time period, indicating if it provides repeatable value.
Importance: It helps determine if the feature is essential to users' workflows or if they can't live without it. Metrics include daily, weekly, or monthly active users (DAU, WAU).
Potential Questions:
- For each user group using the feature, how often are they using it?
- Are they using the feature as intended?
- How much time do users spend within the product?
- Where in the user flow are they using the feature?
3. Depth
Definition: Depth measures the level of engagement users have with a particular feature, reflecting how much it matters to them.
Importance: It looks at the number of product features used by the average person or account, indicating overall product engagement. For example, with Spotify, are users just playing songs, or are they also curating playlists and sharing music?
Potential Questions:
- What percentage of daily, weekly, or monthly active users use this specific feature?
- What is the ratio of DAU/WAU to MAU?
- What percentage of users actively engage with 3 or more features?
- What is the frequency of engagement within specific user groups?
4. Usability
Definition: Usability measures the effort required for users to accomplish goals with your feature, assessing task completion speed and success.
Importance: It helps identify friction points in user flows. Measuring completion rate or using Effort Analysis is recommended.
Potential Questions:
- What percentage of users who start a flow finish it (Completion rate)?
- What is the total number of interactions to complete a task?
- How much time does it take to move between interactions?
5. Sentiment
Definition: Sentiment captures how customers feel when engaging with a product, a more qualitative metric.
Importance: Features that establish positive emotional connections often lead to superior retention rates. Tracking sentiment requires qualitative methods.
Potential Approaches:
- Develop follow-up plans with customer success for NPS survey responses.
- Ask promoters for reasons behind their good scores.
- Meet with detractors to understand improvement areas.
Sentiment can be tracked via customer interviews, external reviews, support tickets, and NPS ratings, often requiring partnership with customer success teams.
Conclusion and Next Steps
Start by using the provided worksheet to track metrics relevant to your feature or product. Expect surprises, as results may differ from imagination. This is a crucial first step towards data-informed decision-making.
As you identify critical metrics for product release success, set measurable targets before development and use results to refine goals. Further guidance is available in subsequent guides.
Congratulations on taking the first step towards becoming a more insights-driven organization!
About Heap
Heap is the premier system of insight for digital experience builders. Its mission is to illuminate hidden opportunities for fast-moving digital teams, enabling them to delight customers and drive business impact at scale by improving key metrics, increasing revenue, enhancing conversion, and accelerating decision-making. Over 8,000 businesses use Heap.
Visit heap.io to learn more.
Heap - Better Insights. Faster. | Heap