Choose product metrics by starting with the customer behavior that represents received value, then connect it to business contribution and the product mechanism. Define population, event, window, and exclusions precisely. Use a primary measure, diagnostic indicators, and guardrails rather than a large dashboard where any movement can be called success.
Who this is for: Product managers and analysts defining how a product, feature, or strategic outcome will be evaluated after release.
- Begin with a value model and causal chain before selecting convenient events from the analytics catalog.
- Write an operational definition that another analyst can reproduce from the same raw data.
- Balance one primary outcome with mechanism diagnostics and guardrails for quality, trust, and unequal effects.
Model value before measurement
Describe how the target customer receives value and how that behavior supports the product's business model. A project tool may create value when a team coordinates real work, not when one person opens the dashboard. Subscription value may connect repeated successful coordination to retention, while support and infrastructure costs affect viability.
Draw the causal chain from product capability through exposure, action, customer result, repeat use, and business effect. Metrics should illuminate this chain. Avoid selecting monthly active users merely because it is familiar. Activity can grow while the meaningful job remains unfinished or while customers are forced to return because the workflow is confusing.
Define a primary outcome
Choose one measure close to the intended customer outcome and sensitive enough for the team's decision. Specify entity, eligible population, numerator, denominator, event properties, observation window, time zone, identity resolution, and exclusions. 'Weekly collaboration rate' is incomplete until everyone agrees what collaboration and eligibility mean.
Set a baseline and meaningful target based on strategy, customer consequence, historical variation, and feasible influence. A target should not be reverse-engineered solely from an executive aspiration. Document whether the metric is a rate, count, duration, distribution, or cohort measure because each reveals and hides different behavior.
Add diagnostics and guardrails
Diagnostic metrics test the mechanism: exposure, setup completion, first use, repeat use, and funnel transitions. They help explain why the primary outcome moved or did not move. Leading indicators are useful for speed but need validation against later value. Remove measures that nobody would use to make a different decision.
Guardrails identify unacceptable tradeoffs such as errors, unwanted messages, refunds, latency, accessibility failures, support burden, or concentration of harm in one group. Establish thresholds and owners. A product should not meet its success definition by making cancellation harder, creating accidental sharing, or shifting hidden work to customer support.
Validate and govern the metric
Test event firing, duplicate handling, late data, identity merges, bot traffic, and source reconciliation. Compare dashboards with raw records and known scenarios. Annotate launches and tracking changes. If the metric definition changes, preserve versions and avoid presenting the new series as directly comparable without explanation.
Review metrics with qualitative evidence and segments. Ask who is missing, what behavior the number cannot explain, and whether teams are gaming the target. Assign a metric owner and review cadence. Retire measures that no longer support decisions instead of adding each new request to a permanently expanding dashboard.
Define success for shared report comments
A reporting product adds comments so analysts and decision makers can resolve questions beside a chart.
- Model value as a report question receiving an informed response before the related decision deadline.
- Define the primary rate using eligible comment threads, a qualifying reply, and a seven-day resolution window.
- Add exposure, first comment, reply time, and repeat participation as mechanism diagnostics.
- Set notification opt-outs, unresolved harassment reports, permission leaks, and support contacts as guardrails.
- Validate events with test accounts and review thread samples alongside the quantitative rate.
Metric definition sheet
Complete this record before placing a metric on a product scorecard.
- Value link: customer job, value event, business contribution, causal role, and decision supported.
- Definition: entity, population, numerator, denominator, event, window, identity, exclusions, and source.
- Benchmark: baseline, natural variation, target, smallest meaningful movement, and review period.
- Companion set: mechanism diagnostics, lagging confirmation, quality checks, and harm guardrails.
- Governance: owner, validation tests, segments, version history, dashboard location, and retirement rule.
Common mistakes
- Using a raw engagement count as success without showing that the action creates customer value.
- Leaving denominator and eligibility undefined so teams calculate different rates under one label.
- Adding dozens of secondary metrics and selecting whichever one supports the desired launch story.
Try one
A meditation app chooses daily screen time as its main success metric. Critique it and suggest a better measurement structure.
A good answer notes that more screen time may conflict with the customer's goal. It proposes a value event such as completing an intended practice or sustaining a chosen routine, with a precise cohort and window. Session starts can diagnose the mechanism, while unwanted reminders, rapid abandonment, and self-reported usefulness can provide guardrail and qualitative context.
Sources
- Atlassian product metrics guideOfficial guidance on product metrics for engagement, retention, and business performance.
- Nielsen Norman Group analytics and user experienceResearch-based guidance on using analytics to identify behavior that needs investigation.