Becoming a Better Analyst: The ABCs of A/B Testing

A/B testing is simple, until it's not! Before you launch your next experiment, make sure you’ve stress-tested the fundamentals: bias, control, timing, and the metrics that truly matter.

“Will it Play in Peoria?” That is a classic saying when it comes to a marketing experiment. Peoria, Illinois was considered the middle of America many decades ago and marketers rushed to this city to test their new products to assess if they should be rolled out nationally.

You don’t hear this saying anymore. Marketing experiments are now known under a number of names, such as, A/B Testing, Champion/Challenger, Marketing Experiments, etc. We will go with A/B Testing for our purposes here. A/B testing is very popular today due to their simplicity of methodology, flexibility of usage, and speed of execution compared to more complex multi-touch attribution or marketing mix models. One can use A/B testing to assess new products, marketing campaigns, pricing strategy or any number of marketing actions.

What happened to “playing it in Peoria?” Competition found out you were testing in Peoria and messed with your experiment, making it no longer viable and unbiased. Today, you can perform a marketing experiment pretty much anywhere, particularly in a digital world.

However, there are some fundamentals when it comes to A/B testing that you should be aware of. At TAP Analytics, we refer to these as the “ABCs of A/B Testing”, or watch-outs, as you design your A/B experiment that you need to be mindful of, and they are as follows:

Bias

Bias in A/B Testing can significantly undermine the credibility of your results

• Cells do not have large enough sample sizes to yield statistically significant results

• Cell populations are not balanced – demographics, geography, behavior, etc. You won’t be able to isolate the impact of the tested variable without interference from other factors.

Time

Time related factors can heavily influence the outcome of your A/B test

• Test period (or pre- / post-) are too short and may not capture the full effect of your marketing action

• Test was conducted in a seasonal period without the proper control impacted by the same seasonal conditions

Control

• You have not designed the right control and can’t really test the specific treatment without worrying about external factors

• You don’t have a proper control for each item you are testing that you want to quantify. Spacing different marketing tests and ensuring proper control for each behavior elevates the trustworthiness of the results.

Measures

• You don’t have the right metrics and your test doesn’t drive any action, or worse, drives a misguided action

Competition

• They did something that affected your test - reduced/increased marketing, altered prices, launched a new product, took a product out of the marketplace, etc.

Other External Factors

• You have not accounted for the economy, weather, etc. without the same control impacted by these factors as well

At TAP Analytics, we regularly coach analytics teams on test design, from statistical soundness to stakeholder-ready storytelling. If you're looking to uplevel your team's experimentation chops, reach out at hello@thetapconsultancy.com.