Practice : A/B Testing
Purpose and Strategic Importance
A/B Testing enables evidence-based decisions by running controlled experiments that compare variants of a feature or experience.
It provides quantitative data on which variant performs better, reducing reliance on opinion or assumption.
Description of the Practice
- Two or more variants are randomly exposed to users.
- Metrics such as conversion, engagement, or retention are tracked.
- Insights guide whether to roll out, adjust, or discard features.
How to Practise It (Playbook)
1. Getting Started
- Formulate a clear hypothesis.
- Define measurable success criteria.
- Split user traffic randomly across variants.
2. Scaling and Maturing
- Run experiments continuously for optimisation.
- Use tools like Optimizely or LaunchDarkly.
- Document results in a shared experimentation log.
3. Team Behaviours to Encourage
- Rigorous use of evidence in decision-making.
- Openness to being proven wrong.
- Sharing results transparently.
4. Watch Out For…
- Declaring results without statistical significance.
- Running too many tests simultaneously.
- Mistaking correlation for causation.
5. Signals of Success
- Qualitative: Teams confident in decisions backed by data.
- Quantitative: Conversion and retention improvements visible post-tests.