Excerpt from free eBook, "The Essential Guide to A/B Testing for Digital Advertisers"
Marketers perpetually ask if test results are “statistically significant,” but what does that term even mean? Probably not what you think.
Traditional statistics relies on p-values as a measure of the “statistical significance” of test results. Most marketers are surprised to learn that p-values do not, in fact, measure the probability that one creative or treatment will outperform another. What p-values measure is far more abstract and removed from the decisions that marketers make based on A/B tests.
In every A/B test one variation will perform at least slightly better than the other. P-values measure the probability that a test result (say, creative variation B outperforms creative variation A by 10%) would have occurred if in fact there were no difference between the two creatives at all. That “95% confidence level” threshold you’ve probably heard bantered about simply means that there is a 5% chance that, were the two variations identical, you would have observed as large a difference in performance between them as you did. The p-value is an important measure in other fields of study to account for what is known as in traditional statistics as Type I error.4In our experience, we have yet to hear a digital marketer ask us for this specific probability. And why would they.
Learn why Adacus uses Bayesian statistics in our latest eBook
"The Essential Guide to A/B Testing for Digital Advertisers"