How to Calculate Testing Lift and Incrementality

There comes a time in every testing practitioner’s life when they have to answer the age-old question – how do I estimate the annualized incremental benefit of the experiment I just ran? To truly understand the answer, we must go back to the beginning, before the test had even met its first unique visitor…

Clear Criteria Set a Foundation to Meaningfully Analyze Test Results

Before an experiment is sent off into the world wide web, it is critical that its creator provides it with clear direction. Without this supervision, an otherwise great idea might get lost on the way to success; its impact might be questioned even if it appears to be a winner. So, it is vital that some guidelines are set before the test begins. At a bare minimum, it needs:

One primary KPI with a relevant baseline completion rate on hand.
Estimated sample size and run-time requirements.

If those two criteria are not set when the test is launched, it becomes difficult to accurately validate whether your test results are statistically significant or not. Several automated tools may try to deceive you along the path to significance too, with confidence levels based merely on a few days of data. Do not fall for this trap! It is best to have your north star identified as precisely as possible before you plot a course for a positive site change.

Measuring Incrementality is Key to Analyzing Test Results

After the journey is complete, all test creators hope that their experiment reached its requirements, and a statistically significant lift was validated through analysis. This is a truly proud parent moment when it happens. But before you go to put it on the refrigerator door, you will want to have some answers ready for when you get asked about it.

“Wow, that is wonderful! What kind of impact will we really see from it though?”

“Can you put this into terms I can better understand?”

Getting caught without an answer to these common questions may again weaken an otherwise positive lift. However, armed with a concrete outcome for your test, you will be well equipped for the next discussion about your testing budget. Not sure how to calculate conversion lift? Look no further – the Merkle|Cardinal Path Annualized Incremental Estimator is here to help make measuring incrementality a breeze.

This quick and easy incrementality calculator makes it effortless to see how many additional conversions you can expect to see per year. Just drop in the dates you ran the test, the audience sizes, and the conversion rates for the winning treatment alongside the control. You will instantly see how many new conversions your test should drive in a given year based on your lift.

Do you have revenue associated with the conversion you lifted? Drop the amount per conversion into the tool and you will also see how much more revenue the test should generate per year. Now you are ready to show off the brilliant work you did!

Try the Annualized Incremental Estimator tool now!