Shadow testing of contextual machine learning models
Shadow testing may be performed to determine whether an updated machine learning model (e.g., an updated machine learning algorithm) is suitable for deployment and replacement of a currently deployed machine learning model. In certain instances, the shadow testing involves machine learning models that determine predictions on the context of language-based user interactions (e.g., the machine learning algorithms are contextual machine learning algorithms). The shadow testing may include asynchronous determinations of predictions by the updated machine learning model on live traffic data being provided to the existing machine learning model. From the shadow testing, disagreement rates between the models and accuracy rates of the models may be used to determine whether the updated machine learning model should be deployed.
1 . A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to:
deploy a first machine learning algorithm for determining one or more first predictions based on classification of contextual information by one or more first classifiers;
implement a second machine learning algorithm for determining one or more second predictions based on classification of contextual information by one or more second classifiers, wherein the second machine learning algorithm is an update of the first machine learning algorithm, the second machine learning algorithm implementing at least one classifier that is an update of the corresponding classifier implemented by the first machine learning algorithm;
receive client data, the client data including contextual information corresponding to language-based interactions with one or more users;
apply the first machine learning algorithm to determine one or more first predictions based on the received client data;
apply the second machine learning algorithm asynchronously from the first machine learning algorithm to determine one or more second predictions based on the received client data;
determine a number of disagreed predictions, wherein the disagreed predictions are predictions for which the second predictions disagree with the first predictions;
determine a disagreement rate for the second machine learning algorithm based on the number of disagreed predictions and a total number of predictions;
deploy the second machine learning algorithm to replace the first machine learning algorithm when the disagreement rate is below a disagreement rate threshold;
determine, for the disagreed predictions when the disagreement rate is above the disagreement rate threshold, a first accuracy rate of the first predictions and a second accuracy rate of the second predictions based on known ground truth values of the disagreed predictions; and
deploy the second machine learning algorithm to replace the first machine learning algorithm when the second accuracy rate exceeds the first accuracy rate by a predetermined amount.
2 . The system of claim 1 , wherein the memory stores further program instructions that when executed further cause the at least one processor to:
determine one or more performance metrics for the first machine learning algorithm;
determine one or more performance metrics for the second machine learning algorithm; and
assess the performance metrics for the first machine learning algorithm and the performance metrics for the second machine learning algorithm to determine whether to deploy the second machine learning algorithm to replace the first machine learning algorithm when either the disagreement rate is below the disagreement rate threshold or the second accuracy rate exceeds the first accuracy rate by the predetermined amount.
3 . The system of claim 1 , wherein the memory stores further program instructions that when executed further cause the at least one processor to:
generate a user interface that displays values for one or more of the disagreement rate, the first accuracy rate, the second accuracy rate, the performance metrics for the first machine learning algorithm, and the performance metrics for the second machine learning algorithm.
4 . The system of claim 1 , wherein the first predictions and the second predictions include one or more of the following types of predictions: speech recognition predictions, intent classification predictions, slot filling predictions, and sentiment analysis predictions.
5 . A method, comprising:
implementing, by a computer system, a second machine learning algorithm for determining one or more predictions based on classification of contextual information by one or more classifiers, wherein the second machine learning algorithm is an update of a first machine learning algorithm currently deployed by the computer system, the second machine learning algorithm implementing at least one classifier that is an update of the corresponding classifier implemented by the first machine learning algorithm;
receiving, by the computer system, client data, the client data including the contextual information corresponding to one or more users;
determining, by the first machine learning algorithm, one or more first predictions based on the received client data;
determining, by the second machine learning algorithm asynchronously from the first predictions, one or more second predictions based on the received client data;
determining, by the computer system, a disagreement rate between the first predictions and the second predictions;
determining accuracy rates for the first machine learning algorithm and the second machine learning algorithm based on the disagreement rate being above a disagreement threshold; and
deploying the second machine learning algorithm to replace the first machine learning algorithm based on an accuracy rate for the second machine learning algorithm exceeding an accuracy rate for the first machine learning algorithm.
6 . The method of claim 5 , further comprising, based on the disagreement rate being above the disagreement rate threshold:
annotating content in the received client data with known ground truth values for predictions that are in disagreement between the first predictions and the second predictions;
determining, by the computer system, the accuracy rate for the first machine learning algorithm based on the annotated content for the predictions that are in disagreement between the first predictions and the second predictions;
determining, by the computer system, the accuracy rate for the second machine learning algorithm based on the annotated content for the predictions that are in disagreement between the first predictions and the second predictions.
7 . The method of claim 6 , further comprising, based on the accuracy rate for the second machine learning algorithm not exceeding the accuracy rate of the first machine learning algorithm, refining the second machine learning algorithm.
8 . The method of claim 7 , wherein refining the second machine learning algorithm includes:
assessing the second machine learning algorithm and the second predictions; and
providing additional training for the second machine learning algorithm based on the assessment.
9 . The method of claim 5 , wherein the contextual information in the received client data includes one or more of the following types of contextual information: text information, audio information, and image information.
10 . The method of claim 5 , wherein the first machine learning algorithm and the second machine learning algorithm are implemented to provide predictions for speech recognition in language-based interactions with the one or more users.
11 . The method of claim 5 , wherein the first machine learning algorithm and the second machine learning algorithm are implemented to provide predictions for intent classification for language-based interactions with the one or more users.
12 . The method of claim 5 , wherein the first machine learning algorithm and the second machine learning algorithm are implemented to provide predictions for slot filling for language-based interactions with the one or more users.
13 . The method of claim 5 , wherein the first machine learning algorithm and the second machine learning algorithm are implemented to provide predictions for sentiment analysis for language-based interactions with the one or more users.
14 . The method of claim 5 , further comprising:
storing the first predictions and the second predictions in a data store as the predictions are determined; and
retrieving, by the computer system, the first predictions and the second predictions from the data store after a predetermined number of predictions have been made to determine the disagreement rate between the first predictions and the second predictions.
15 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
receiving data from language-based interactions with one or more users, the data including contextual information corresponding to the language-based interactions with the one or more users;
operating a first machine learning algorithm for determining one or more first predictions on the received data based on classification of the contextual information by one or more first classifiers, wherein the first predictions are provided as real-time predictions for the language-based interactions;
asynchronously operating a second machine learning algorithm for determining one or more second predictions on the received data based on classification of the contextual information by one or more second classifiers, wherein the second classifiers include least one classifier that has been changed from the first classifiers;
after a predetermined number of first and second predictions have been made, determining a disagreement rate between the first predictions and the second predictions;
switching the second machine learning algorithm to provide the real-time predictions for the language-based interactions when the disagreement rate is below a disagreement rate threshold;
when the disagreement rate is above the disagreement rate threshold, determining accuracy rates for the first machine learning algorithm and the second machine learning algorithm; and
switching the second machine learning algorithm to provide the real-time predictions for the language-based interactions when an accuracy rate for the second machine learning algorithm exceeds an accuracy rate for the first machine learning algorithm.
16 . The one or more non-transitory, computer-readable storage media of claim 15 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement when the disagreement rate is above the disagreement rate threshold:
annotating content in the received data with known ground truth values for predictions that are in disagreement between the first predictions and the second predictions;
determining the first accuracy rate for the first machine learning algorithm based on the annotated content for the predictions that are in disagreement; and
determining the second accuracy rate for the second machine learning algorithm based on the annotated content for the predictions that are in disagreement; and
switching the second machine learning algorithm to provide the real-time predictions for the language-based interactions when the second accuracy rate exceeds the first accuracy rate by a predetermined amount.
17 . The one or more non-transitory, computer-readable storage media of claim 16 , storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
providing additional training for the second machine learning algorithm when the second accuracy rate does not exceed the first accuracy rate by the predetermined amount.
18 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein switching the second machine learning algorithm to provide the real-time predictions for the language-based interactions includes stopping the first machine learning algorithm from providing the real-time predictions for the language-based interactions.
19 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein the disagreement rate threshold is a disagreement rate of at most 10%.
20 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein asynchronously operating the second machine learning algorithm for determining the one or more second predictions includes operation of the second machine learning algorithm to provide predictions for the language-based interactions offline from the one or more users.