7) [Algorithm] Interpreting the results of experiments

When your test ad converts better than your control ad, it doesn’t necessarily mean that it’s the winner.

Example 1) Your test ad converts 20% better than the control ad. But if it’s less than your minimum detectable effect, then you deal with statistical noise.

Example 2) Your test ad converts 30% better than the control ad. But the profit may suffer, the number of leads may drop, and they may even be more expensive. Why? Because it may drive fewer clicks (due to drop in CTR and/or in impression share) and/or much higher CPC.

This lesson is laid out as an algorithm. It helps you decide whether you should claim your test ad as the winner, the loser, or conclude that this is a “no winner” case.


This content is from the Premium section of our Library. It's exclusively available to members with the Premium account.

Buy Premium account


Current Premium member? Sign in.