A/B Test Sample Size Calculator

Creative testing in digital advertising is hobbled by the need for very large sample sizes, making tests expensive and lengthy.  Adacus brings the latest in A/B test statistics to digital marketers with a Bayesian-powered A/B statistics platform.  

Use this calculator to determine the number of impressions you'll need using Adacus to be confident in the results of your A/B test.


What makes the Adacus A/B testing platform different?

Traditional A/B test statistics has held back digital advertising A/B testing because its success metric for statistical significance - p-values - require very large sample sizes when tests have low conversion rates - as is common in digital advertising banner testing.  But statistical significance based on p-values isn't even what digital marketers want to know.  Adacus cuts testing time in half by reporting the metrics you actually need to know, metrics which fall within success thresholds faster than p-values:

  • Chance to Beat Control: Unlike the misleading p-value reported in traditional A/B test platforms, Adacus reports the Chance to Beat of each creative variation.  Adacus reports the Chance to Beat a control and the Chance to Beat all other variations.
  • Potential Loss: You've selected a winning creative variation, and you know there a chance it's a false positive and the losing variation could beat it.  But do you know by how much?  Adacus reports the Expected Loss of selecting a variation as the winner. This tells you how much is at risk if the losing variation turns out to be the winner.   

How do I know when I can stop a test?

Set a threshold for the Chance to Beat and/or the Potential Loss and, when the threshold is met, a winner is declared and the test stopped. Common thresholds for the Chance to Beat in digital advertising are 75%, 80%, 90% and 95%, while common thresholds for Potential Loss are 0.5% 1% and 2%.  Adacus can provide estimated test run times for different thresholds.

Does Adacus provide consulting support on test planning, administration, and interpretation?

Absolutely, we love supporting and empowering digital marketing teams and include unlimited consulting on test planning, administration and interpretation in our contracts.

Have other questions of your own?  Let's get in touch.