Cardinal Path’s response to COVID-19 Cardinal Path is sharing all we know to help marketers during COVID-19.  Learn more.

Back in January, I wrote a post about the dangers of rushing to judgment based on small numbers. I recommended that even if GWO has declared a winner, it's best to let experiments continue to run at least two weeks and gather at least 100 conversions per version. Today I'm going to play devil's advocate and argue that in some cases, you might just want to pull the plug earlier!

We're currently running an A/B test for a lead-generation website. We believed that the original version of the page had lots of room for improvement, so we were pretty confident we could boost conversions.

GWO very quickly confirmed our suspicions: four days into the test, GWO declared a winner. Our version B was outperforming the original by 139%.


We urged the client to keep the test running, for the reasons discussed in my earlier post. They agreed and the test is still running. In the past few days, the observed improvement has fluctuated, but it's clear the new page is better. It's just a question of how much better.

My normal inclination would be to keep the test running until the numbers settle down. But there is a serious potential downside to doing so: by continuing to show the losing version to half of our client's visitors, our client is potentially losing sales. The longer we keep the test running, the more our client is potentially losing!

So… do we keep the test running until we get more precise numbers? Or do we stop the test now, take full advantage of the improved performance of the new page, and move on to the next test?

I'd like to suggest some guidelines as to when it's better to end experiments earlier than normally recommended. For example, if all these criteria are met, perhaps it's better to stop the experiment:

  • GWO has declared a winner.
  • The results, though early, indicate a very large difference in performance between pages;
  • There is no reason to doubt the early results (i.e. the large performance difference is not unexpected);
  • There is no reason to expect that seasonal or day-of-week/month factors may have skewed results; and
  • Each conversion has a substantial monetary value (i.e. there's a good chance that keeping the experiment running is costing the client money).

It seems to me, the bottom line is this: though GWO utilizes the scientific methodologies of A/B and multivariate testing, its purpose is for marketing, not pure science. We need to know which page performs better, but we don't necessarily need to know precisely how much better.

Keeping an experiment running can be costly. Sometimes it's better just to pull the plug early and move on to the next test.

Sharing is caring!

Popular

COVID-19 Crisis Navigator​

In partnership with Dentsu, Cardinal Path helps you distill the overwhelming news and information into a bi-weekly report highlighting emerging trends and insights during the pandemic.

EXPLORE THE REPORT

Analytics Rising Logo

FREE VIRTUAL EVENT

October 14

Practical Strategies Delivered By Top Minds in Digital Marketing

Register for #AnalyticsRising and get a Starbucks Gift Card when you share the news.

Message Sent

Thank you.

Click here to download access the tool.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Success!
Your message was received.

Thank you.

Message Sent

Thank you for registering.

Thank you for submitting the form.

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Thank you for your submission.

Message Sent

Thank you for registering.

Thank you for registering.​

Paid media spend by Government websites increased a whopping 139% YoY in 2020.

2020 Online Behavior Live Dashboard

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

2020 Online Behavior Live Dashboard

Message Sent

Thank you for registering.

Thank you for submitting the form.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Thank you for registering.

Cardinal Path is continuing with its series of free training. Next we are conducting training on Google Data Studio. Check it out here.
Cardinal Path hosted a live session to connect with you and answer all your questions on Google Analytics.
Get all the expertise and none of the consultancy fees in this not-to-be-missed, rapid-fire virtual event.

Message Sent

Success! Thank you
for reaching out.