Back in January, I wrote a post about the dangers of rushing to judgment based on small numbers. I recommended that even if GWO has declared a winner, it’s best to let experiments continue to run at least two weeks and gather at least 100 conversions per version. Today I’m going to play devil’s advocate and argue that in some cases, you might just want to pull the plug earlier!

We’re currently running an A/B test for a lead-generation website. We believed that the original version of the page had lots of room for improvement, so we were pretty confident we could boost conversions.

GWO very quickly confirmed our suspicions: four days into the test, GWO declared a winner. Our version B was outperforming the original by 139%.

We urged the client to keep the test running, for the reasons discussed in my earlier post. They agreed and the test is still running. In the past few days, the observed improvement has fluctuated, but it’s clear the new page is better. It’s just a question of how much better.

My normal inclination would be to keep the test running until the numbers settle down. But there is a serious potential downside to doing so: by continuing to show the losing version to half of our client’s visitors, our client is potentially losing sales. The longer we keep the test running, the more our client is potentially losing!

So… do we keep the test running until we get more precise numbers? Or do we stop the test now, take full advantage of the improved performance of the new page, and move on to the next test?

I’d like to suggest some guidelines as to when it’s better to end experiments earlier than normally recommended. For example, if all these criteria are met, perhaps it’s better to stop the experiment:

  • GWO has declared a winner.
  • The results, though early, indicate a very large difference in performance between pages;
  • There is no reason to doubt the early results (i.e. the large performance difference is not unexpected);
  • There is no reason to expect that seasonal or day-of-week/month factors may have skewed results; and
  • Each conversion has a substantial monetary value (i.e. there’s a good chance that keeping the experiment running is costing the client money).

It seems to me, the bottom line is this: though GWO utilizes the scientific methodologies of A/B and multivariate testing, its purpose is for marketing, not pure science. We need to know which page performs better, but we don’t necessarily need to know precisely how much better.

Keeping an experiment running can be costly. Sometimes it’s better just to pull the plug early and move on to the next test.

Message Sent

Thank you for registering.

Cardinal Path hosted a live session to connect with you and answer all your questions on Google Analytics.
Get all the expertise and none of the consultancy fees in this not-to-be-missed, rapid-fire virtual event.

Thank you for submitting the form.

Thank you for submitting the form.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Click here to download access the tool.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Success!
Your message was received.

Thank you.

Thank you for registering.

Cardinal Path is continuing with its series of free training. Next we are conducting training on Google Data Studio. Check it out here.

Message Sent

Thank you for registering.

Thank you for your submission.

Your request has been submitted and a rep will reach out to you shortly.

Message Sent

Thank you for your interest.

Thank you for registering.

You should receive a confirmation email from GoToWebinar with your unique webinar login information. If you do not receive this email or have trouble logging in to the event, please email asmaa.mourad@cardinalpath.com.

Thank you for subscribing!

You're now looped into the world's largest GMP resource hub!

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Thank you for registering.​

Paid media spend by Government websites increased a whopping 139% YoY in 2020.

2020 Online Behavior Live Dashboard

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

2020 Online Behavior Live Dashboard

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Success! Thank you
for reaching out.