In this post, a member of the Cardinal Path’s Data Science team, Danika Law, explains why selecting the right error metric for your business problem is so crucial. In the early project phases of running a dashboard or predictive model, we always address the client’s needs first and foremost. This post explains why, to a data scientist, selecting the right error metric is  a necessary step to ensure accuracy of the model and solve for the problem at hand.

When you are building a predictive model, no matter what problem you are trying to solve for, you should always specify your error metric right from the start of the forecasting project, and make sure that it is in line with your business goals. Some error metrics punish overestimates more than underestimates, while others punish larger errors exponentially. Having the errors punished according to what most makes sense for your business will lead to a better model.

There should be one main consideration when choosing your error metric: what is the cost associated with making an incorrect prediction, and where is this cost the largest?

  • Do we want to punish overestimates or underestimates more heavily?
    • For example, when classifying churners (those customers who are going to drop off), if the cost of misclassifying other customers as potential churners is less than the cost of missing some actual churners, we will want to punish underestimates more heavily. This helps ensure we don’t miss any churners.
  • Are there segments which have greater costs associated with incorrect predictions?
    • For example, let’s say we are classifying customers into “High Value Customer”, “Medium Value” and “Low Value” based on the predicted lifetime revenue they will generate for the business. If we misclassify someone as a “High Value Customer” and then spend a lot of advertising resources on them and end up not being a high value customer, the return on investment will be low. It is for this reason that the value needs to be weighted accordingly.
  • Should we punish larger errors or smaller errors more heavily?
    • For example, let’s say we are forecasting revenue by marketing channel. The predictions will be used to help plan what to expect, so having very high or very low projected revenue that is incorrect could lead to making poor business decisions. On the other hand, being off by a small amount here and there doesn’t change the decisions made to such a large degree. In cases like this, we’d want to punish large errors more heavily than small errors.

Once you have specified how you want your error metric to punish errors, you’ll need to select one that fits your business needs. Here are some resources that will help you select:

Ensuring that you specify your business problem correctly, and choose the correct error metric will help ensure that the cost of making a mistake is balanced out by how you evaluate your model.

Message Sent

Thank you for registering.

Cardinal Path hosted a live session to connect with you and answer all your questions on Google Analytics.
Get all the expertise and none of the consultancy fees in this not-to-be-missed, rapid-fire virtual event.

Thank you for submitting the form.

Thank you for submitting the form.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Click here to download access the tool.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Success!
Your message was received.

Thank you.

Thank you for registering.

Cardinal Path is continuing with its series of free training. Next we are conducting training on Google Data Studio. Check it out here.

Message Sent

Thank you for registering.

Thank you for your submission.

Your request has been submitted and a rep will reach out to you shortly.

Message Sent

Thank you for your interest.

Thank you for registering.

You should receive a confirmation email from GoToWebinar with your unique webinar login information. If you do not receive this email or have trouble logging in to the event, please email asmaa.mourad@cardinalpath.com.

Thank you for subscribing!

You're now looped into the world's largest GMP resource hub!

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Thank you for registering.​

Paid media spend by Government websites increased a whopping 139% YoY in 2020.

2020 Online Behavior Live Dashboard

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

2020 Online Behavior Live Dashboard

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Success! Thank you
for reaching out.