Prognostics and health management service
Systems, methods, and apparatuses for selecting a model are described. A method includes receiving a request to perform model selection and evaluating multiple models by generating various metrics for each trained model. These metrics include a forewarning time (how much in advance of a failure an alert can be raised), an event recall metric (how many failure events were alerted to in advance), an event precision metric (ratio of true and false positives, and an area under a receiver operating characteristic (ROC) curve. For each trained model, a weighted harmonic mean is calculated from these metrics. A model is then selected based on the calculated means and a report on the selected model is generated and provided.
1. A computer-implemented method comprising:
receiving, at a model selector of a prognostics and health management service of a multi-tenant provider network, a request to perform model selection, wherein the multi-tenant provider network receives sensor data from one or more managed devices;
receiving, at the model selector, labeled time-series data sets;
training, by the model selector, a plurality of models based on the labeled time-series data sets to yield a plurality of trained models;
evaluating, by the model selector executing model selection code using one or more processors, the plurality of trained models to select a trained model by:
applying, as input, a common set of testing data to the plurality of trained models to generate, as output from the plurality of trained models, a plurality of metrics for each of the plurality of trained models, the plurality of metrics comprising all of:
a forewarning time metric of how much in advance of a failure an alert can be raised,
an event recall metric of how many failure events were alerted to in advance of failure,
an event precision metric of a ratio of true and false positives, and
an area under a receiver operating characteristic (ROC) curve,
calculating, for each of the plurality of trained models, a weighted harmonic mean of all of the plurality of metrics generated for the trained model, wherein the area under the ROC curve generated for the trained model is assigned a greater weight when calculating the weighted harmonic mean than the event recall metric generated for the trained model, and wherein the event recall metric generated for the trained model is assigned a greater weight when calculating the weighted harmonic mean than both the event precision metric generated for the trained model and the forewarning time metric generated for the trained model, and
selecting one of plurality of trained models based on the calculated weighted harmonic means;
causing a prediction to be made, using the selected model, of a remaining useful life of a managed device of the one or more managed devices;
causing, in response to the prediction, maintenance to be performed on the managed device;
generating and providing a report regarding the selected one of the plurality of trained models;
receiving implicit user feedback including a number of discarded alerts; and
reevaluating the plurality of trained models, based on the implicit user feedback, to select a different trained model, the different trained model is different from the selected one of the plurality of trained models.
2. The computer-implemented method of claim 1 , wherein the request includes one or more of:
an identifier of a location of an initial labeled time-series data set;
an identifier of a location of testing data;
an identifier of a location to store the selected one of the plurality of trained models;
one or more identifiers of one or more users allowed to provide feedback;
an identifier of what weights to use in the selection of a machine learning model; or
one or more identifiers of one or more users allowed to receive anomaly data.
3. The computer-implemented method of claim 1 , further comprising:
receiving explicit user feedback regarding the labeled time-series data sets;
updating the labeled time-series data sets to generate updated labeled time-series data sets; and
based on receiving the explicit user feedback, retraining the plurality of models or the plurality of trained models based on the updated labeled time-series data sets.
4. A computer-implemented method comprising:
receiving, at a model selector of a prognostics and health management service of a multi-tenant provider network, a request to perform model selection, wherein the multi-tenant provider network receives sensor data from one or more managed devices;
evaluating, by the model selector executing model selection code using one or more processors, a plurality of trained models to select a trained model by:
applying, as input, a common set of testing data to the plurality of trained models to generate, as output from the plurality of trained models, a plurality of metrics for each of the plurality of trained models, the plurality of metrics comprising all of:
a forewarning time metric of how much in advance of a failure an alert can be raised,
an event recall metric of how many failure events were alerted to in advance of failure,
an event precision metric of a ratio of true and false positives, and
an area under a receiver operating characteristic (ROC) curve,
calculating, for each of the plurality of trained models, a weighted harmonic mean of all of the plurality of metrics generated for the trained model, and
selecting one of the plurality of trained models based on the calculated weighted harmonic means;
causing a prediction to be made, using the selected model, of a remaining useful life of a managed device of the one or more managed devices;
causing, in response to the prediction, maintenance to be performed on the managed device;
generating and providing a report regarding the selected model, wherein the report comprises the forewarning time metric, the event recall metric, the event precision metric, and the area under the ROC curve;
receiving implicit user feedback including a number of discarded alerts; and
reevaluating the plurality of trained models, based on the implicit user feedback, to select a different trained model, the different trained model is different from the selected model.
5. The computer-implemented method of claim 4 , wherein the request includes one or more of:
an identifier of a location of an initial labeled time-series data set;
an identifier of a location of testing data;
an identifier of a location to store the selected one of the plurality of trained models;
one or more identifiers of one or more users allowed to provide feedback;
an identifier of what weights to use in the selection of a machine learning model; or
one or more identifiers of one or more users allowed to receive anomaly data.
6. The computer-implemented method of claim 4 , wherein the forewarning time is a mean fraction.
7. The computer-implemented method of claim 4 , wherein the event precision is calculated by dividing a number of true positives by a sum of the number of true positives and a number of false positives in a time window.
8. The computer-implemented method of claim 4 , further comprising:
training a plurality of models based on labeled time-series sensor data sets to yield the plurality of trained models;
receiving explicit user feedback regarding the labeled time-series sensor data sets;
updating the labeled time-series data sets based on the explicit user feedback to generate updated labeled time-series data sets; and
retraining the plurality of models or the plurality of trained models based on the updated labeled time-series data sets.
9. The computer-implemented method of claim 4 , further comprising:
receiving implicit user feedback regarding usage of alerts by a receiver indicating false positive detections; and
based on receiving the implicit user feedback, reevaluating the plurality of trained models to select a different trained model, the different trained model is different from the selected model.
10. The computer-implemented method of claim 4 , further comprising probing interaction with a user regarding usage of a model by raising an alert to attempt to force feedback.
11. The computer-implemented method of claim 10 , wherein the feedback includes one or more of new labels, weights, or training data.
12. The computer-implemented method of claim 4 , wherein the report includes, for a labeled anomalous range, whether the anomalous range was detected and/or with how much forewarning time the anomalous range was detected.
13. The computer-implemented method of claim 4 , wherein the report includes false positives as ranges and durations.
14. The computer-implemented method of claim 4 , further comprising:
receiving labeled time-series sensor data; and
training a plurality of models based on the labeled time-series sensor data to yield the plurality of trained models.
15. A system comprising:
one or more electronic devices to implement a model selection service in a multi-tenant provider network, wherein the multi-tenant provider network receives sensor data from one or more managed devices, the model selection service including instructions that upon execution by one or more processors cause the model selection service to:
receive, at the model selection service, a request to perform model selection,
evaluate, by the model selection service executing model selection code using the one or more processors, a plurality of trained models to select a trained model by:
applying, as input, a common set of testing data to the plurality of trained models to generate, as output from the plurality of trained models, a plurality of metrics for each of the plurality of trained models, the plurality of metrics comprising all of:
a forewarning time metric of how much in advance of a failure an alert can be raised,
an event recall metric of how many failure events were alerted to in advance of failure,
an event precision metric of a ratio of true and false positives, and
an area under a receiver operating characteristic (ROC) curve,
calculating, for each of the plurality of trained models, a weighted harmonic mean of all of the plurality of metrics generated for the trained model, and
selecting one of the plurality of trained models based on the calculated weighted harmonic means to use to manage the first one or more electronic devices,
cause a prediction to be made, using the selected model, of a remaining useful life of a managed device of the one or more managed devices;
cause, in response to the prediction, maintenance to be performed on the managed device;
generate and provide a report regarding the selected model, wherein the report comprises the forewarning time metric, the event recall metric, the event precision metric, and the area under the ROC curve;
receive implicit user feedback including a number of discarded alerts; and
reevaluate the plurality of trained models, based on the implicit user feedback, to select a different trained model, the different trained model is different from the selected model.
16. The system of claim 15 , wherein the request includes one or more of:
an identifier of a location of an initial labeled time-series data set;
an identifier of a location of testing data;
an identifier of a location to store the selected one of the plurality of trained models;
one or more identifiers of one or more users allowed to provide feedback;
an identifier of what weights to use in the selection of a machine learning model; or
one or more identifiers of one or more users allowed to receive anomaly data.
17. The system of claim 15 , wherein the model selection service further comprises instructions that upon execution cause the model selection service to:
train a plurality of models based on labeled time-series sensor data sets to yield the plurality of trained models;
receive explicit user feedback regarding the labeled time-series sensor data sets;
update the labeled time-series data sets based on the explicit user feedback to generate updated labeled time-series data sets; and
retrain the plurality of models or the plurality of trained models based on the updated labeled time-series data sets.
18. The system of claim 15 , wherein the model selection service further comprises instructions that upon execution cause the model selection service to:
receive implicit user feedback regarding usage of alerts by a receiver indicating false positive detections; and
reevaluate, responsive to receiving the implicit user feedback, the plurality of trained models to select a different trained model, the different trained model is different from the selected model.
19. The system of claim 15 , wherein the model selection service further comprises instructions that upon execution cause the model selection service to probe interaction with a user regarding usage of a model by raising an alert to attempt to force feedback.
20. The system of claim 19 , wherein the feedback includes one or more of new labels, weights, or training data.