Tests with statistically insignificant results lead to wrong decisions and money loss. The more ads you want to test, the higher the budget you need to get statistically significant results.
So, how do you define:
- Exactly how many ads you can test in what specific number of days for the specific budget you have
- What conversion event you should use as an optimization event for the budget you have
- An exact number of clicks you need to accumulate to be able to claim an ad as the winner or loser (or conclude that your experiment has no winner)
- Whether you are keeping a losing ad active for too long
- Whether you are concluding that an ad is a winner too soon
There are dull formulas to calculate each of these values. Instead, I’ve prepared something neat for you.
In this lesson, you’ll find a simple table for decision making. It tells you: what conversion event to optimize for, how many ads at a time, and how many ads per month you should test for your budget to get statistically significant results. Either you have 100 visitors/day or 100,000 visitors/day – you’ll find values for your case.