Standard : A/B Test Coverage

Description

A/B Test Coverage measures the percentage of user-facing changes that are tested through controlled experiments before full rollout. It helps ensure decisions are driven by evidence rather than intuition.

Higher coverage suggests teams are systematically testing hypotheses rather than relying solely on opinion.

How to Use

What to Measure

• Number of changes shipped behind an A/B test or controlled rollout.
• Total number of user-facing changes released in the same period.

Formula

A/B Test Coverage (%) = (Number of Tested Changes ÷ Total User-Facing Changes) × 100

Example: 20 of 50 changes tested via A/B → 40% coverage.

Instrumentation Tips

• Use feature flags to enable experimentation tracking.
• Track test creation and completion in analytics tools.
• Define what counts as a “testable” change (UI, pricing, flow).

Why It Matters

• Evidence-based decision-making: Reduces risk of negative impact.
• Incremental improvement: Supports data-driven optimisation.
• Customer trust: Minimises harmful or confusing changes.

Best Practices

• Automate test setup and analysis where possible.
• Use holdout groups to measure lift accurately.
• Sunset tests and clean up flags promptly.

Common Pitfalls

• Testing insignificant changes, creating noise.
• Not reaching statistical significance before deciding.
• Treating tests as optional for major changes.

Signals of Success

• Coverage rises over time without slowing delivery.
• Increased win rate of tested changes.
• Fewer negative customer impacts post-release.

Related Measures

• [[Experiment Success Rate]]
• [[Learning Velocity]]
• [[CoE/Agile/Measures/Value Realisation/Feature Adoption Rate]]