How to Choose the Right KPI?

Image result for product scorecardProduct key performance indicators (KPIs) are metrics that measure your product’s performance. They help you understand if the product is meetings its business goals and if the product strategy is working. Without KPIs, you end up guessing how your product is performing.

Long vs short term metrics

  • KPI/Business metrics, counter metrics
  • Optional (deep dive/debug/perf/safety metrics)
  • Segmentation (comparing to competitors by segments)
  • Time Frame

Image result for product scorecard

1 State the Business Goals of your Product

To choose the right key performance indicators or KPIs, you must be clear on the business goals your product serves. If your product directly generates revenue, then revenue is likely to be a key indictor, for example. If you are not sure what the business goals are, then ask yourself how the product benefits the company and why the business invests in it.

The Product Vision Board

2 Make the Goals Measurable

Knowing the business goals of your product is a prerequisite for selecting the right KPIs. But it is not enough. To effectively apply the indicators, analyse the resulting data, and take the right actions, the goals must be measurable. The challenge is to establish measurable goals that are also realistic, particularly for brand-new and young products. The next tips helps you address this challenge.

3 Use Ratios and Ranges

Work with ratios and ranges to quantify your goals, as Alistair Croll and Benjamin Yoskovitz recommend in their book Lean Analytics. Instead of stating that a new product should create x amount of revenue per year, you could say that the product should increase the company’s revenue by five to 10% within one year after its launch, for instance. While this goal might still be unrealistic, I have at least drawn a line in the sand against which progress can be measured. If it turns out that the goal is too ambitious, then you should recognise this: move the line and adjust your target.

4 Avoid Vanity Metrics

Stay clear of vanity metrics, measures that make your product look good but don’t add value. Take the number of downloads for an app as an example. While a fair amount of people might download the product, this tells you little about how successful it is. Instead of measuring downloads, you should choose a relevant and helpful metric, such as daily active usage or referral rate.

5 Don’t Measure everything that can be Measured

Don’t measure everything that can be measured and don’t blindly trust an analytics tool to collect the right data. Instead, use the business goals to choose a small amount of metrics that truly help you understand how your product performs. Otherwise you take the risk of wasting time and effort analysing data that creates little or no insights. In the worst case, you action irrelevant data and make the wrong decisions.

6 Use Quantitative and Qualitative KPIs

As their name suggests, quantitative indicators, such as daily active users or revenue, measure the quantity of something rather than its quality. This has the benefit of collecting “hard” and statistically representative data. Qualitative indicators, such as user feedback, help you understand why something has happened, for instance, why users aren’t as satisfied with the product as expected. Combining the two types gives you a balanced outlook on how your product is doing. It reduces the risk of loosing sight of the most important success factor: The people behind the numbers, the individuals who buy and use the product.

7 Employ Lagging and Leading Indicators

Lagging indicators, such as revenue, profit, and cost, are backward-focused and tell you about the outcome of past actions. Leading indicators help you understand how likely it is that your product will meet a goal in the future. Take product quality as an example. If the code is becoming increasingly complex, then adding new features will become more expensive and require more time. Meeting profit targets and delivery dates will therefore become harder. Using backward and forward-focused indicators allows you to tell you if you have met the business goals and helps you anticipate if the product is likely to meet the goals in the future.

8 Look beyond Financial and Customer Indicators

Financial indicators, such as revenue and profit, and customer metrics, including engagement and referral rate, are the two most common indicator types in my experience. While these metrics are undoubtedly important, they are not sufficient. Say your product is meeting its revenue and profit goals and that customer engagement and referral rate are high. This suggests that your product is doing well and that there is no reason to worry. But if at the same time, the team motivation is low or if the code quality is deteriorating, then you should be concerned: These indicators suggest that achieving product success will be much harder in the future. You should therefore look beyond financial and customer indicators and complement them with the relevant product, process, and people indicators.

9 Leverage Trends

Compare the data you report to other time periods, user groups, or competitors, such as revenue growth over the last six weeks or cancellation rates from quarter to quarter. This helps you spot trends, for instance, if revenue is increasing, staying flat, or declining. Trends allow you to better understand what’s happening and to take the right actions. If a decline in venue is a one-off occurrence, for instance, then there is probably no reason to be overly worried. But if it is a trend, then you should investigate how you can stop and revert it – unless you are about to sunset your product.

10 Use a Product Scorecard

Once you have selected the right key performance indicators for your product, you should collect the relevant data and regularly analyse it. A product scorecard is a great tool for this job. A good scorecard uses the right indicators and helps you spot trends. Take a look at my product scorecard template below that offers a holistic outlook on the product performance and displays product, process, and people KPIs in addition to financial and customer indicators. You can download the template from and by clicking on the image.

Image result for product scorecard

Characteristics of a metric

  1. Sensitivity and Robustness: Whether the metric is sensitive to changes you care about, and is robust to changes you don’t care about (e.g. mean is sensitive to outliers, median is robust but not sensitive to changes to small group of users). This can be measured by using prior experiments to see if the metric moves in a way that intuitively make sense. Another alternative is to do A/A tests to see if the metric picks up any spurious differences. At Google, it was observed that the analytical estimates of variance was often under-estimated, and therefore they have resorted to use empirical measurements based on A/A test to evaluate variance. If you see a lot of variability in a metric in an A/A test, it is probably too sensitive to be used. Rather than do several multiple A/A tests, one way is to do a large A/A test, and then do bootstrap to generate small groups and test the variability.With A/A tests, we can
    1. Compare result to what you expect (sanity check)
    2. Estimate variance empirically and use your assumption about the distribution to calculate confidence
    3. Directly estimate confidence interval without making any assumption of the data
  2. Distribution: Obtained by doing a distribution on the retrospective data

4 categories of metrics

  1. Sums and counts (e.g. # of users who visited a page)
  2. Distributional metrics – means, medians, percentiles
  3. Probabilities and rates
  4. Ratios

To calculate the confidence interval, you need

  • Variance (or standard deviation)
  • Distribution

For a binomial distribution, your estimated variance is p(1p)/Np(1−p)/N. Estimated variance of the mean is σ2/Nσ2/N. If the underlying data is normal, then the median will be normal. But if the underlying data is not normal, the median may not be normal. The mean is typically normally distributed irrespective of the distribution of the underlying data due to Central Limit Theorem. Difference in counts may be normally distributed. The variance of the difference will be the sum of the variances of each of the 2 counts. Rates tend to have a poisson distribution, and the variance for a poisson is just the mean. For ratios of test over control, both the mean and the variance depends on the distribution of the test and control metrics.

Comments & Responses

Leave a Reply

Your email address will not be published. Required fields are marked *