Google Website Optimizer Hint: Beware of Small Numbers, Don't Rush to Judgment

Whenever we’re about to launch a new experiment, I always give clients the same speech. It goes something like this:

“You probably won’t be able to resist logging into GWO and checking the results every day. But always bear in mind that experiments cannot yield reliable results until they have run long enough to gather sufficient data. Early results are often surprising — even alarming — but you must resist jumping to conclusions. Let the experiment run its course.”

Now I have a confession to make: I sometimes have a very hard timing taking my own advice. A recent example comes to mind.

We were running an A/B test of a lead-generation form for a Financial Institution. Our revised (“B”) version quickly took the lead over the original. Within a week or so, it was showing an improvement of about 200%! We were astonished, and of course we started congratulating ourselves on how clever we were.

Luckily, we didn’t share our internal boasting with the client. Because over the next few weeks, we watched in dismay as our lead fell to 150%… to 100%… to 50%… and eventually settled at around 20%. Still respectable, but imagine how bad we’d have looked if we had told the clients we’d given them a 200% lift!

In other cases, we’ve seen GWO declare a winner, only to “undeclared” it the next day. You can’t rely entirely on what the tool is telling you.

So when can you confidently say the experiment has run its course? I’d suggest a good rule of thumb is, when all three of these criteria are met: