Google Website Optimizer's Newest Feature: Automatic Pruning

GWO just keeps getting better and better. About a year ago, Google introduced one of GWO’s most useful features: the ability to prune losing combinations from experiments. The benefits were clear:

Low-performing combinations no longer hurt overall conversion rates.
Experiments could complete faster.

The only danger is that you must be cautious in using this feature. Once you prune a combination, you can’t put it back in. So if you prune out of “panic” based on insufficient data, you might actually prune a potential winner. Deciding whether or not to prune is sometimes an agonizing decision.

Now, Google has taken the pruning functionality a step farther and removed the necessity of making agonizing decisions. With “auto-disabling”, losing combinations are automatically removed from your experiment. You can even adjust how aggressively your experiments will be pruned:

Conservative: Red performance bar “distinctly separated” from the original’s bar.
Moderate: Red performance bar just starting to separate from the original’s.
Aggressive: Performance bar still yellow, but only a bit of overlap.

Google hasn’t stated the exact degrees of certainty the tool uses, but I’d guess that conservative is about 99%, moderate is 97.5% and aggressive is 95%? (I’d welcome clarification/confirmation here…)

This is a very useful new feature. Though there will be times when I’d prefer to remain in control, auto-disabling can potentially make the experimental process much simpler. If nothing else, it will save a lot of emails and phone calls in which we debate which combinations to prune, and when.

What’s next? Well, there may be reasons why you want certain combinations to remain in your experiment even if their performance drops below the others. So perhaps the ability to exempt combinations from auto-pruning would be a useful feature.