There are a variety of things we can test with A/B testing ranging from;
to less user visible backend changes like;
However, A/B testing has also its limitations and there are some cases where it might not be the best or most straightforward tool to use. Here are the two most prominent use cases:
1. If we want to understand why users are acting in a certain way: With A/B testing, we can see the changes in the quantitative metrics, understand which version performs better and by how much, but it is not possible to understand “why” since it doesn’t give us explanations.
To have a more holistic picture and provide details and depth for understanding the users’ behaviors, needs and motivations, we should make sure that we are always combining qualitative and quantitative insights and not just settling for the answers of the questions who, what, when, and how but also looking for why.
2. If we want to measure long-term effects: In A/B testing, it is straightforward to measure the short-term effect, i.e. the impact observed during the experimentation period. However, there are some scenarios where the short-term effect is not always predictive of the long-term effect. For example, raising prices likely increases short-term revenue but also likely reduces long-term revenue as users abandon.
Although, there might be three possible approaches to use A/B testing to measure long term effects:
However, since neither of these options are straightforward, the best would be focusing on changes with quick effects which we can measure in a short term if we don’t have the setup and the tools which enables us to measure long term effects.
We’ll be discussing how much change is appropriate in A/B tests next time. So, stay tuned!
A/B testing course by Google:
Controlled Experiments on the Web: Survey and Practical Guide: