Model performancemonitoring is a way to ensure deployed models are sustainable andperforming optimally over time. Often models are built and deployed and the system performance, like model failures and records processed, aremonitored to ensure the model is running. While this type of monitoring is important, system monitoring cannot tell if the model is performing as it was intended or is beginning to decay. Model performance monitoring goes deeper, evaluating the model inputs and outputs to detect more subtle signals that a model may not be delivering its intended value before the model starts to impact key business decisions.
Causes of Model Performance Decay
Future Data Variances
Actual data may vary from the data set the model was built on
Change in Behavior
Changing policies, trends, and technological advances can change behavior over time
Events like the Covid-19 pandemic can have a sizable impact on consumer behavior
Changes to Internal Processes
System upgrades and process changes that seem small may have a large effect
External events can have a dramatic effect on consumer behavior. Covid 19 is a great example of that. Businesses and offices closed, shopping behavior changed, and people stopped traveling. The repercussions for predictive models was often very dramatic. There are ways to mitigate the damage to your models. Read more in this articled, entitled “The Covid 19 Butterfly Effect”.