A/B Testing

Which ad will attract more people? Will a specific shape of a button make you more likely to click? Knowing the answers to these questions are of vital importance to firms as it increases their revenue. At first, you may think that the firms would have to go by their intuition and just hope they are lucky. However, everyone’s minds work differently, and so we can’t always go by our intuition. Sometimes, we have to simply test the two (or more) options and see which performs best. This is A/B testing.

Context & Examples

A/B testing is not a very recent method; it’s been used for quite a while now - virtually as well as in real life. However, non-digital testing is a much harder process than digital testing. When experiments are done offline, it can take a very long time and is often quite expensive.

For example, if we wanted to test the effect of teachers’ attendance rates on students’ average scores, we would first need two different environments. In the first environment, we would want the teachers’ attendance rates to be quite low, and in the second environment quite high.

Now, we can’t just tell teachers to have a low attendance rates as there would be many ethical concerns here - we don’t want to potentially ruin students’ scores just for an experiment. Thus, for the first environment, we would have to find a school in which teachers’ attendance rates are already low. This can take quite a while. Additionally, the teachers who are in the second environment would want an incentive to have a high attendance rate, perhaps a financial reward. This is what makes it expensive.

Furthermore, we wouldn’t get our results in a day or two; this experiment can take many months or even years to obtain significant findings. This is an example of A/B testing offline.

Digital A/B Testing

Experiments can also be done digitally, and these are much easier, cheaper and faster to run. Say we want to test which wording of a title attracts more clicks. Here, all we need to do is write some code to simulate the test. First, we would divide our visitors into two groups. We show the first group the first style of wording, and the second group the other style. If group A had a higher click percentage, then we would show future users the style shown to group A.

Applications

These types of experiments take a very short time to conduct, and have just about no cost. Due to the ease, companies like google are running A/B testing all the time. For instance, they would test between 2 (or 41) different shades of blue to see which is the most attractive to viewers. Of course, many times, option A is only slightly better than option B, but sometimes the results can be astonishing. Option A may potentially be 600% better than option B, which is what keeps companies constantly doing these tests.