Challenger Models

Challenger models is a way to enable the testing of different models that address the same problem in a production environment. This feature is particularly useful when you want to test out a new variation of a model (or a retrained version) for sometime before actually using it. That way you'll know exactly how the model will behave in a production environment since it will have been already running there for some time.

You can think of challenger models as deployed models that are running "in the background" with your live model. Challenger models receive the same requests sent to the live model and are expected to produce output of the same format. All the metrics monitored on the live model are also available for challenger models to enable you to compare performance differences and take action based on that.

After deploying a model and its challenger(s), you may notice that a challenger model is performing better than the current live model. In that case, you can "promote" the challenger model to be the deployment's live model, i.e., it will be the model chosen to respond to the prediction requests for the deployment. The previous live model will be added as a challenger.