PROCESSING MODEL OUTPUTS
This disclosure describes techniques for capturing data points from a collection of models. In one example, this disclosure describes a method that includes capturing a sequence of model output data generated by a model, wherein the sequence of model output data includes information about a plurality of predictions made by the model; selecting, based on configuration settings, a process to perform on the sequence of model output data, wherein the process is selected from a plurality of available processes; performing the selected process, based on at least a portion of the sequence of model output data, to generate information about performance of the model over time; and sending, to a downstream system, control signals to modify operation of the downstream system based on the information about performance of the model over time.
1 . A method comprising:
capturing, by a computing system, a sequence of model output data generated by a model, wherein the sequence of model output data includes information about a plurality of predictions made by the model, and wherein each prediction in the plurality of predictions is generated by the model in response to a different set of model input data;
selecting, by the computing system and based on configuration settings, a process to perform on the sequence of model output data, wherein the process is selected from a plurality of available processes;
performing the selected process, by the computing system and based on at least a portion of the sequence of model output data, to generate information about performance of the model over time; and
sending, by the computing system and to a downstream system, control signals to modify operation of the downstream system based on the information about performance of the model over time.
2 . The method of claim 1 , wherein the model is a first model, wherein the model input data is first model input data, wherein the sequence of model output data is a sequence of first model output data, wherein the plurality of predictions is a first plurality of predictions, and wherein the method further comprises:
capturing, by the computing system, a sequence of second model output data generated by a second model, wherein the sequence of second model output data includes information about a second plurality of predictions made by the second model, wherein each prediction in the second plurality of predictions is generated by the second model in response to a different set of second model input data; and
performing the selected process, by the computing system and based on at least a portion of the sequence of second model output data, to generate information about performance of the second model over time.
3 . The method of claim 2 , wherein sending the control signals includes:
sending control signals to modify operation of the downstream system further based on the information about performance of the second model over time.
4 . The method of claim 2 , wherein the downstream system is a first downstream system, and wherein sending the control signals includes:
sending control signals to modify operation of a second downstream system based on the information about performance of the second model over time.
5 . The method of claim 1 , wherein the selected process includes assessing accuracy of the model, and wherein the method further comprises:
sending, by the computing system and to a business unit computing system, alerts about model inaccuracies.
6 . The method of claim 1 , wherein the selected process includes assessing accuracy of the model, and wherein sending the control signals includes:
sending control signals to model retraining infrastructure to cause the model retraining infrastructure to retrain the model.
7 . The method of claim 1 , wherein the selected process includes performing analytics on the model output data, wherein the method further comprises:
sending, by the computing system and to a business unit computing system, near-real time business intelligence reports.
8 . The method of claim 1 , wherein the selected process includes performing analytics on the model output data, and wherein sending the control signals includes:
sending control signals to a downstream computing system that responds to the control signals by modifying operation of a production system.
9 . The method of claim 8 , wherein sending control signals to the computing system further includes:
enabling the downstream computing system to cause the production system to change how the production system performs at least one of: monitoring for fraud, fulfilling online sales orders, processing loans, processing loan applications, or selecting an advertisement.
10 . The method of claim 1 , wherein the selected process includes monitoring health of the model, and wherein performing the selected process includes:
identifying an underperforming aspect of the model.
11 . The method of claim 10 , wherein sending the control signals includes:
sending control signals to model remediation infrastructure to cause the model remediation infrastructure to remediate the underperforming aspect of the model.
12 . The method of claim 1 , wherein the selected process includes performing load balancing of resources used by a production system, and wherein sending the control signals includes:
sending control signals to adjust, based on predictions made by the model, allocations of resources used by the production system.
13 . The method of claim 1 , wherein the selected process includes performing load balancing of resources used by the computing system, and wherein sending the control signals includes:
sending control signals to adjust, based on predictions made by the model, allocations of resources used by the computing system.
14 . A computing system comprising processing circuitry and a storage device, wherein the processing circuitry has access to the storage device and is configured to:
capture a sequence of model output data generated by a model, wherein the sequence of model output data includes information about a plurality of predictions made by the model, and wherein each prediction in the plurality of predictions is generated by the model in response to a different set of model input data;
select, based on configuration settings, a process to perform on the sequence of model output data, wherein the process is selected from a plurality of available processes;
perform the selected process, based on at least a portion of the sequence of model output data, to generate information about performance of the model over time; and
send, to a downstream system, control signals to modify operation of the downstream system based on the information about performance of the model over time.
15 . The computing system of claim 14 , wherein the model is a first model, wherein the model input data is first model input data, wherein the sequence of model output data is a sequence of first model output data, wherein the plurality of predictions is a first plurality of predictions, and wherein the processing circuitry is further configured to:
capture a sequence of second model output data generated by a second model, wherein the sequence of second model output data includes information about a second plurality of predictions made by the second model, wherein each prediction in the second plurality of predictions is generated by the second model in response to a different set of second model input data; and
perform the selected process, based on at least a portion of the sequence of second model output data, to generate information about performance of the second model over time.
16 . The computing system of claim 15 , wherein to send the control signals, the processing circuitry is further configured to:
send control signals to modify operation of the downstream system further based on the information about performance of the second model over time.
17 . The computing system of claim 15 , wherein the downstream system is a first downstream system, and wherein to send the control signals, the processing circuitry is further configured to:
send control signals to modify operation of a second downstream system based on the information about performance of the second model over time.
18 . The computing system of claim 14 , wherein the selected process includes assessing accuracy of the model, and the processing circuitry is further configured to:
send, to a business unit computing system, alerts about model inaccuracies.
19 . The computing system of claim 14 , wherein the selected process includes assessing accuracy of the model, and wherein to send the control signals, the processing circuitry is further configured to:
send control signals to model retraining infrastructure to cause the model retraining infrastructure to retrain the model.
20 . Non-transitory computer-readable media comprising instructions that, when executed, cause processing circuitry of a computing system to:
capture a sequence of model output data generated by a model, wherein the sequence of model output data includes information about a plurality of predictions made by the model, and wherein each prediction in the plurality of predictions is generated by the model in response to a different set of model input data;
select, based on configuration settings, a process to perform on the sequence of model output data, wherein the process is selected from a plurality of available processes;
perform the selected process, based on at least a portion of the sequence of model output data, to generate information about performance of the model over time; and
send, to a downstream system, control signals to modify operation of the downstream system based on the information about performance of the model over time.