Testing & Personalization – Moving from Conversion Rates to ROI

If you have been following our blog, you know that we love the field of testing and personalization. Not only does experimenting give us opportunities to glean interesting insights into human behavior, but it also allows us to help our clients grow through the provision of more optimized user experiences. 

There’s a lot of buzz around the benefits of testing and personalization. And of course, we agree with the positive impact experiments can have, but we also know that it’s not always rainbows and unicorns.

One of the challenges we’ve observed is the hyperfocus on conversion rates. It’s not surprising to see this considering virtually all digital tests map back to some kind of conversion rate. But focusing on conversion rates can end up being a trap – this can be explained with the law of diminishing returns.

The law of diminishing returns explained

If you’re not familiar with the law of diminishing returns, it can essentially be summarized by this – the more effort you put into something (e.g. time, money), at some point in time, the less you get out of it.  

It’s interesting that it seems to apply to many things in life – from business to personal decisions. In the testing and personalization world, it simply means that the more you test something, the rate at which conversion rates increase will begin to decrease.

Why might this happen?

Testing and personalization activities are typically focused on promotional aspects of marketing – experimenting with different features of a marketing asset in an effort to persuade users to engage in a specific action (e.g. calls to action, design elements, content layout key messaging).

Referring back to fundamental marketing principles, there are other considerations that could impact the likelihood that someone will engage in your desired action, like product, price or place considerations (anyone remember the 4 Ps?).

It’s simple really – if you build a product no one wants, for example, you will likely not continue to see a drastic increase in conversion rates, regardless of how many tests you run.

Timing is everything

It is true that as marketers, we really don’t know when the law of diminishing returns will start to occur. However, we can begin to identify it by simply mapping our conversion rates over time. We can do this for a specific marketing asset (e.g. a webpage), user flow (e.g. shopping cart), or overall testing program.  

Consider the following chart. In this particular example, there were some great results early on, but as testing activities have increased over time, conversion rates have continued to increase, but at a declining rate – that’s the law of diminishing returns.

At some point in time – this will happen.

But the type of test matters…

If that’s what you are thinking – agreed. But, while it’s possible that testing strategies are playing a role in the flattening out (or worse, decline!) of conversion rates, by mapping conversion rates over time, you should be able to see a trendline that reflects a similar pattern – whether it be for a specific test or a testing program.    

At some point in time, if all other marketing initiatives (e.g. product strategies, pricing strategies) stay constant, the growth in conversion rates from testing an asset or an overall testing program will begin to flatten out.

What happens when we start getting diminishing returns?

There are a number of options available in the testing world. For instance, if the observation is being made for a specific marketing asset, like a webpage, test a different marketing asset.  

If you are noticing diminishing returns occurring at a program level, it may have to do with the types of tests being used. It is common for companies to start with split testing and notice they are getting diminishing returns from those types of tests and they start evolving into more complicated testing strategies like personalization.

But before making those decisions – ensure you are doing your best to calculate your return on investment.

Deprioritizing conversion rates

Whether you continue on a particular testing path really should come down to business impact. Conversion rates are one thing (and are the focal point of virtually all digital tests), but the monetary impact is much more telling.

For instance, consider the following hypothetical data from two different websites:

Table 1 – Website A Test

Group	Number of Visits	Conversion Rate	Revenue per Conversion	Total Revenue
Control Group	10,000	3.00%	$45	$13,500.00
Experimental Group	10,000	6.00%	$45	$27,000.00

Group	Number of Visits	Conversion Rate	Revenue per Conversion	Total Revenue
Control Group	10,000	3.00%	$100	$30,000.00
Experimental Group	10,000	3.50%	$100	$35,000.00

Table 2 – Website B Test

Now take the summary data that also includes hypothetical costs:

Website	Conversion Rate Increase	Incremental Revenue Increase	Test Cost	ROI
Website A	100.00%	$13,500	$5,000	170%
Website B	16.67%	$5,000	$1,000	400%

In summary, we can see that Website B’s test had higher ROI even though it had a lower conversion rate – something that will occur as the law of diminishing returns starts to kick in.

Conclusion

If you are just focusing on conversion rates for your tests or overall testing program, you may start to make decisions that impact future testing decisions. Don’t ignore them, but deprioritize them in favor of ROI – while this may sound obvious to some, in practice, it is not as common as you think.