When is A/B testing helpful?

“Almost any question can be answered cheaply, quickly and finally, by a test campaign. And that’s the way to answer them, not by arguments around a table.”
— Claude Hopkins

There are a variety of things we can test with A/B testing ranging from;

• New features,
• Changes to the user interface (UI),

to less user visible backend changes like;

• Ranking, recommendation or search algorithms,
• Speed or latency.

However, A/B testing has also its limitations and there are some cases where it might not be the best or most straightforward tool to use. Here are the two most prominent use cases:

1. If we want to understand why users are acting in a certain way: With A/B testing, we can see the changes in the quantitative metrics, understand which version performs better and by how much, but it is not possible to understand “why” since it doesn’t give us explanations.

To have a more holistic picture and provide details and depth for understanding the users’ behaviors, needs and motivations, we should make sure that we are always combining qualitative and quantitative insights and not just settling for the answers of the questions who, what, when, and how but also looking for why.

2. If we want to measure long-term effects: In A/B testing, it is straightforward to measure the short-term effect, i.e. the impact observed during the experimentation period. However, there are some scenarios where the short-term effect is not always predictive of the long-term effect. For example, raising prices likely increases short-term revenue but also likely reduces long-term revenue as users abandon.

Although, there might be three possible approaches to use A/B testing to measure long term effects:

• Defining metrics measurable in the short-term which can predict the long-term (proxy metrics),
• Analyzing the effects months later after stopping the test by keeping the user identifier to see if there is a long-term effect,
• Running tests for longer periods of time.

However, since neither of these options are straightforward, the best would be focusing on changes with quick effects which we can measure in a short term if we don’t have the setup and the tools which enables us to measure long term effects.

We’ll be discussing how much change is appropriate in A/B tests next time. So, stay tuned!
References:
A/B testing course by Google:
https://www.udacity.com/course/ab-testing–ud257

Controlled Experiments on the Web: Survey and Practical Guide:
https://exp-platform.com/Documents/controlledExperimentDMKD.pdf