The Blog

A/B Testing with Google Analytics Content Experiments

Google Analytics has announced a new A/B testing feature called Content Experiments. This is a pretty significant evolutionary step for Google Analytics in making it an analytics and optimization tool. Think of this as Google Website Optimizer being baked right into the Google Analytics interface. Using Content Experiments in lieu of GWO will allow you to easily define content URLs and goals for your experiments, analyze your reports more efficiently and will eliminate the need for all those extra GWO tracking codes on your site.

I want to review the basics of A/B testing and running GA Content Experiments, as well as discuss some important technical details and advanced considerations. For existing GWO users it’s very important that you get familiar with GA Content Experiments because the standalone GWO product will be unavailable after August 1, 2012.

What is A/B Testing?

A/B testing takes a lot of forms. For this, we are specifically talking about A/B page testing. You define a control page (page A) and a variation (page B) of that original page to test against.  The purpose of this test is to expose your audience to the different versions of a page to determine which version will result in more conversions for your site.

A/B testing is very easy (and free) with Google Website Optimizer. Read how we helped YouTube increase signups by 15.7% through conversion optimization.

How to Setup a Content Experiment

Google Analytics Content Experiments - start page

You can access Content Experiments by logging into your Google Analytics account, opening the profile you want to run an experiment in and click the Standard Reporting tab. In the left menu click Content, then Experiments, then start experimenting!

The Google documentation for Content Experiments is pretty thorough so I don’t want to duplicate the info here, but I want to briefly describe the Content Experiment process and provide some additional commentary.

  1. Prepare. It’s very important that you identify the business objectives of your site, then setup GA goals to align with those objectives. Then you will need to put some good thought into what to test. Identify poor performing pages by reviewing your Landing Page reports for pages with high bounce rates, or your Page reports for pages with unusually high exit rates. Then hypothesize on what can be done to improve the performance of those pages. This is what you’ll want to base your experiment on.
  2. Configure & Modify. After you’ve identified your control page and created the variation pages, you’ll need to provide the appropriate URLs in the experiment setup. At this time a single content experiment can support up to 5 variation pages in addition to your control page. You will then configure some additional experiment options (such as identifying the GA goal you’re trying to improve) and grab the experiment code that will need to be placed in the <head> of your original page.  Read the detailed instructions on setting up an experiment.
  3. Track Progress and Stopping an Experiment. The reporting screen for an experiment will show you the number of visits, conversions and conversion rate that each experiment page has contributed to. An experiment can end via a few different methods: a) GA determines a winning page; b) The experiment has expired after running for 3 months; or c) You’ve manually stopped the experiment.
  4. #winning. If your experiment is able to identify a winner, congrats! If the winner was a variation page, you may want to consider replacing your control page with the winning version.

After you’ve completed an experiment, start the process again with another page and keep trying to make incremental improvements to your site!

Important Things to Consider

Content Experiments Graph

Be patient. Context Experiments will not choose a winning variation until an experiment has run for at least 2 weeks. This is a good thing. I can’t tell you how often I’ve seen erratic conversion activity in the opening days of a test. Conversion trends usually take a few weeks to stabilize. It won’t matter if you have enough of a visitor sample size within your 1st day. Day to day and week to week activity is so varied. Just wait a few weeks.

Calculate your sample size estimations. Although you can configure up to 5 variation pages for an experiment, be aware that adding a variation will require more visitors to meet your sample size requirements and will increase the time your experiment needs to run for.  Use our sample size calculator tool to determine the proper sample sizes for your experiment and how long you can expect a test to run for before determining a winner.

Be very careful when running multiple concurrent experiments. The danger is that if you have visitors from one experiment that interact with elements of another experiment running on your site, it’s very difficult to account for these interactive effects.  This will skew your data and lead to some faulty conclusions. Even though you can setup 12 experiments in a profile, that doesn’t mean that you should run 12 experiments at once. Unless you’re absolutely confident in how to analyze multiple concurrent experiments, it’s okay be safe and run one experiment at a time.

Content Experiment redirects are safe. What actually happens in Content Experiment A/B testing is that when a visitor visits your control page and GA instead shows a variation page to the visitor – it will immediately perform a JS redirect to the variation page. With GWO this commonly resulted in some wacky referrals showing up in your analytics data unless you reconfigured your analytics code.  Content Experiments has resolved this issue automatically by passing over the ‘utm_referrer’ query string parameter that tells GA what the true referrer is.  And in terms of SEO, there shouldn’t be any concern. Although I believe that search bots are pretty good with processing JavaScript, remember that Google Analytics is a JS and cookie-based solution, which many bots don’t typically handle very well. In addition, the Google search team is very aware of Google-based tracking/testing codes and will preserve your original pages as they should be.

URL Parameters pass through on redirects.  If somebody visits your original page via a URL with extra query string parameters (such as GA campaign tags), those parameters will be preserved and visible in the redirected URL.

Handling dynamic content. In the past I’ve always used GWO multivariate experiments to handle testing dynamic templates like product display pages. We don’t have that option, though, with Context Experiments (more on that later). If your dynamic content is served via query string parameters, you can rely on the previously mentioned passthrough of URL parameters to render your pages appropriately.  But if your dynamic content is served via permalink-style URLs, there isn’t an easy way to do this currently with Content Experiments. Let’s keep hoping for Content Experiments MV testing in the future!

What About Multivariate Testing?

As of right now, Google Analytics Content Experiments only supports A/B testing and not multivariate testing. With GWO shutting down after August 1, this may (temporarily) eliminate any MV testing with a free Google tool. This is unfortunate since I’m personally a big fan of MV testing. The extra flexibility that comes with that mode allowed me to test anything regardless of website architecture hurdles. There are plenty of other tools out there that are also great for MV testing, but I sure would like MV testing integrated with my Google Analytics profiles.  Fingers crossed!

There are really so many opportunities and considerations with Google Analytics Content Experiments. Do you have any questions or opinions about the tool? Leave a comment or let me know on Twitter. (@adrianvender)

This entry was posted in Technology, User Experience, Web Analytics and tagged , . Bookmark the permalink.
  • http://twitter.com/joseuonline Jose Uzcategui

    Thanks for the post Adrian. This is major :)

  • Geoff Hoffman

    Great post, Adrian. You mentioned utm_referrer above; I wanted to add that a new utm variable is used to pass the experiment ID on the query string also. So, don’t be alarmed if you notice utm_expid={experiment_id_here} in your URLs when implementing content experiments. More here.

  • http://twitter.com/heyglenns Glenn Schmelzle

    Thanks Adrian, you & the Cardinal Path are always up on the changes Google continually rolls out.

    • adrianvender

      Thanks Glenn. It’s always exciting to learn about Google’s latest offerings :)

  • http://gunjanaggarwal.blogspot.com/ Gunjan

    Great post, Adrian !!!

  • Yehoshua Coren

    Hi Adrian,

    Great post.  I especially like that you mentioned that GA has fixed the referrer problem that used to exist in GWO and the issue surrounding dynamic content.  I haven’t had a chance to roll up my sleeves and dive into Content Experiments yet, so it is nice to see some answers to these questions from a trusted source.  :)

    Two questions that came up for me when reading your post.

    1).  You mentioned that there isn’t a “easy way” to do template wide testing on product pages etc where the products have all have static URLs.  Is there a “hard way?”  :)  Since MVT tests aren’t available we can’t apply use any of the redirect hacks that squeezed into GWO that normally were used for template wide testing.  But if template wide testing is really “offline” for a while with Content Experiments, this seems like a major setback for companies.  Am I on target? GWO’s  MVT functionality went far beyond just testing multiple page elements…

    2).  Do custom variables need to be set to integrated into experients so we can use Advanced Segments like we did for GWO, or will we ultimately be able to choose experiment variations as a segment (haven’t seen it yet).   I see that the utm_expid can be pushed to the URL so segments can be created that way, but I prefer to not mess-up content reports.  Thoughts?

    Best regards,

    Yehoshua

    • http://www.convert.com/ Convert.com Experiments

      He Yehoshua, your right this move of Google is great to bring A/B testing to the masses. I think that was the idea. But killing MVT hurts professionals like Adrian and yourself. So I hope you love the trial you just got on yesterday of Convert Experiments. If you need a hand you got my calendar tungle and have me on Skype. I am available when you need a hand.

      Dennis

  • http://www.project-noen.de/ Marco

    perfect for something I’ve been waiting

  • Max Lehmann

    Very informative and well explained article that will be useful to many. 

  • Jhammer

    What are some of your recommended Multivariate Testing tools?

    • http://www.convert.com/ Convert.com Experiments

      May I pitch Convert.com J Hammer? ;-) MVT, A/B and split testing.
      Try is

      Dennis

  • http://www.convert.com/ Convert.com Experiments

    Adrian, I love Multivariate testing and we actually launched Convert Experiments last week with seamless Google Analytics integration… and guess what? Visual A/B testing, Split URL testing … and MVT!

    Try it out at http://convert.com (yep I am the founder so forgive the pitch). But little things like live-stats, one code, multiple goals (that give GA custom variables), revenue tracking and testing… well it will make your live great at as a tester…

    We got a 30 day free trial and love to show you around in the tool to give a high level overview… will I see you soon Adrian?
    Dennis

    • Mahi

      Convert.com might be good but not FREE. That is disappointing.

      • http://www.convert.com/ Convert.com Experiments

        Hi Mahi, I understand completely but Google keeps part for it free so that is still an option. We offer a new option for more professional and faster testing with more insights… so premium tools have a price tag :-)

        Dennis

      • http://www.convert.com/ Convert.com Experiments

        Hi Mahi,

        I went into the depth on what the NEW Google Website Optimizer has in features and how do they compare to the old version. Take a look at this http://blog.convert.com/dirty-details-on-google-website-optimizers-new-big-brother.html

        Dennis

  • Christine

    Thank`s. Shall I put the experiment code in the if I can`t put it in the ? (because I can change for all pages, but can`t  change it for a page). Will it be a mistake or will not?

  • Christine

    Thank`s. Shall I put the experiment code in the body if I can`t put it in
    the head ? (because I can change head for all pages, but can`t  change it for a
    page). Will it be a mistake or will not?

    • http://www.facebook.com/silvio.wagner.33 Silvio Wagner

      Hi Christine. If u r running one test at a time only, u could paste the code into ur general header. The redirects will b configured in ur GA account. It’s not best practise but it works.

  • Thomas Rochford

    If a visitor wants to go to a page that you are A/B testing twice, does this mean that they will go to two different variants or always the same one? I ask , as depending on the variants it could be confusing for the visitor.

    • http://www.facebook.com/silvio.wagner.33 Silvio Wagner

      Hey Thomas. Most people will see the same variation as long as they use the same client and browser an don’t delete cookies or run in private mode. Analytics drops a cookie that shows the same variation to known clients. Hope that helped.

  • SN

    Are we allowed to use the original page url with query parameters as the experimental url? This way we wouldn’t have to create new urls. Does anyone know if this works?

    • Adnan Siddiqi

      use htaccess? :)

  • Marius

    I’ve recenlty launched my first experiment with Content Experiment. All fine except that since then the goal I selected as conversion in the test in not tracked anymore. I haven’t found anything on this in their help center. Do you have any ideas what could have caused the problem?

  • http://twitter.com/ajaxjones ajax jones

    cant believe they stopped the MVT , managed to get a min 5% upsell on every site, and up to 42% on some sites. Maxymiser is too expensive, so we have baked our own. I guess they dropped it as most people never used it or could work out how to do it. 99.9% of analytics are amateurs (in the nicest sense) so probably just too confusing. And they dont get paid for its use so why run it

  • Sharna Davis

    I set up an experiment and tested it but all of my links which were pointing to different pages on my site just redirected to my homepage page again. Any advice? Thanks.

    • http://forex.firepips.com FirePips

      I got exactly the same problem. So far haven’t found any solution.

      • http://twitter.com/NgoMinhNam Nam Ngo

        Make sure to only embed the Content Experiment JS on the pages you want to test.

      • http://twitter.com/NgoMinhNam Nam Ngo

        Make sure to only embed the Content Experiment JS on the pages you want to test.

  • Ryan Masterson

    Thanks for the article. Google Analytics is very powerful.

  • Hillary Wyon

    Is there any documentation on how the new Experiments works with a CDN? Are there any caching concerns that need to be considered when using the new experiments?

  • Thomas O

    I’m running an experiment now with 2 variations, but it is showing the original version a lot more than the variation, meaning the data isn’t 50-50. How can I ensure that the experiment is displaying the original and variation equally amongst the visitors on the site?

    • Cam

      The 50% you’ve set is actually the amount of traffic that gets entered into the experiment, which means if you do a test with a control and a variation, half of that 50% will go to the control, and half will go to the variation. That means only 25% of traffic will go to your variation.

      • http://allanwhite.tumblr.com/ Allan White

        So, in order to test say, a home page variation with the entire audience, the value would be 100%, right?

  • http://www.getdiscountcoupon.com/ Kathy J. Lowrey

    Hi

    Very nice article. It helped me a lot understanding content experiments.

    But I have a question.

    My goal type is currently URL Destination.
    And we push conversions manually to analytics after phone call matching ususally it takes one day.

    We use this code for pushing conversions.

    var _gaq = _gaq || [];
    _gaq.push(['_setAccount', 'XX-XXXXXXXX-X']);
    _gaq.push(['_setAllowAnchor', true]);
    _gaq.push(['_trackPageview','/call/conv ']);

    What I want to know is how do I push a conversion manually for a specific variation.

    Thanks in advance for help.

  • Fabi

    Very helpful article. Thank you!

  • http://www.facebook.com/Chrystalness14 Chris Diggs

    The experiment seems to redirect all my traffic to my versions B and C but never to the orginal is it normal ?

  • http://twitter.com/joanojr John Noren

    Great write up, thanks!

  • htt

    I have been running an experiment for 80 days and want to stop it, but when I click “stop experiment” it says “Experiment Landing page not stopped” Any tips? The experiment seems to be slowing my page a lot. Anyone else had this problem?

  • http://twitter.com/AndrewGraft Andrew Graft

    Our website is built in MODX and a number of the pages are templates that include the same code after the head tag. In my case it means that if I add the Content Experiment code to one page (e.g., the “original” page), it may also appear on the other (the “experiment” page).

    I know Google says the experiment code only needs to appear on the original page, but will it affect the experiment if it also appears on the experiment page? Thanks for any help you can provide.


Cardinal Path Training

Copyright © 2014, All Rights Reserved. Privacy and Copyright Policies.