Model prediction confidence utilizing drift

1 . An Information Handling System (IHS), comprising:

a processor; and

a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution, cause the IHS to:

identify drift with respect to a trained Artificial Intelligence (AI) or Machine Learning (ML) model, based, at least in part, on both a pre-model analysis of at least a variance of input data, and a post-model analysis of an inference result, wherein identify drift further comprises:

provide at least a portion of the input data to a plurality of AI/ML drift detection models;

receive respective drift confidence values from the plurality of AI/ML drift detection models;

select at least one AI/ML drift detection model that provided a smallest drift confidence value among the respective drift confidence values; and

tag the input data with a characteristic label associated with the selected AI/ML drift detection model, wherein the characteristic comprises a hidden feature not detectable in the input data;

calculate a weighted average of a first metric from the pre-model analysis and a second metric from the post-model analysis, wherein a first weight is applied to the first metric, a second weight is applied to the second metric, and at least one of the first weight or the second weight is user-configurable;

adjust a confidence score of a prediction produced by the AI or ML model based, at least in part, on the weighted average of the first and second metrics;

modify the prediction based, at least in part, upon the identified drift; and

in response to a determination that drift exceeds a threshold, indicate an associated prediction is not trusted and initiate a pipeline that comprises corrective retrain, test, and deployment of a revised model based at least in part on a selected subset of tagged input data.

2 . The IHS of claim 1 , wherein the trained AI or ML model comprises at least one of: a Linear Regression model, a Deep Neural Network model, a Logistic Regression model, a Decision Tree model, a Linear Discriminant Analysis model, a Naive Bayes model, a Support Vector Machines model, a Learning Vector Quantization model, a K-nearest Neighbors model, or a Random Forest model.

3 . The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to perform the pre-model analysis based, at least in part, upon calculation of the variance of input data during inference with respect to an input data norm, and wherein the input data norm is established during training in absence of drift.

4 . The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to perform the post-model analysis based, at least in part, upon a prediction or inference norm established for output data from the AI or ML model during training, and output data from the trained AI or ML model during inference, wherein the program instructions, upon execution, further cause the IHS to calculate the second metric based, at least in part, upon a variance of prediction or inference results output from the AI or ML model during inference with respect to the prediction or inference norm, and wherein the prediction or inference norm is established during training in absence of drift.

5 . The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to adjust the confidence score proportionally to the drift.

6 . The IHS of claim 1 , wherein in response to a determination that the trained AI or ML model is more susceptible to data drift than concept drift, the first weight is greater than the second weight.

7 . The IHS of claim 1 , wherein in response to a determination that the trained AI or ML model is more susceptible to concept drift than data drift, the second weight is greater than the first weight.

8 . The IHS of claim 1 , wherein in response to a change of the at least one of the first weight or the second weight, the program instructions, upon execution, further cause the IHS to further adjust the confidence score.

9 . The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to train the AI or ML model based, at least in part, on training data that comprises sensor data that has data characteristics, attributes, or tags identified in discrete instances of selected class types within the training data, wherein the training data comprises at least a subset of the tagged input data selected based on the characteristic associated with the selected AI/ML drift detection model.

10 . The IHS of claim 1 , wherein to adjust the confidence score, the program instructions, upon execution, further cause the IHS to reduce the confidence score proportionally to the drift in response to a determination that the drift is greater than a threshold value.

11 . The IHS of claim 1 , wherein to adjust the confidence score, the program instructions, upon execution, further cause the IHS to indicate that the prediction or inference be trusted in response to a determination that the drift is smaller than a threshold value.

12 . The IHS of claim 1 , wherein to adjust the confidence score, the program instructions, upon execution, further cause the IHS to increase the confidence score proportionally to the drift in response to a determination that the drift is smaller than a threshold value.

13 . The IHS of claim 1 , wherein to adjust the confidence score, the program instructions, upon execution, further cause the IHS to enable a user to select by how much the drift affects the confidence score.

14 . The IHS of claim 1 , wherein the program instructions, upon execution, further cause the IHS to determine a plurality of inferences produced by the AI or ML model based on sensor data from a respective plurality of sensor devices, wherein modify the prediction further comprises individually adjust each inference of the plurality of inferences based, at least in part, on a location of a respective sensor device of the respective plurality of sensor devices, and wherein the program instructions, upon execution, further cause the IHS to:

determine an observation overlap between at least two sensor devices based at least in part on respective device properties;

identify a consensus among inferences produced from sensor data collected within the observation overlap; and

tag input data that give rise to inferences that match the consensus with a ground-truth label, wherein the selected subset of tagged input data is further selected based at least in part on the ground-truth label.

15 . A hardware memory device having program instructions stored thereon that, upon execution, cause an Information Handling System (IHS) to:

determine a plurality of inferences produced by a trained Artificial Intelligence (AI) or Machine Learning (ML) model based on sensor data from a respective plurality of sensor devices;

identify drift with respect to the trained AI or ML model, based, at least in part, on both a pre-model analysis of at least a variance of input data, and a post-model analysis of the plurality of inferences, wherein identify drift further comprises:

provide at least a portion of the input data to a plurality of AI/ML drift detection models;

receive respective drift confidence values from the plurality of AI/ML drift detection models;

select at least one AI/ML drift detection model that provided a smallest drift confidence value among the respective drift confidence values; and

tag the input data with a characteristic label associated with the selected AI/ML drift detection model, wherein the characteristic comprises a hidden feature not detectable in the input data;

calculate a weighted average of a first metric from the pre-model analysis and a second metric from the post-model analysis, wherein a first weight is applied to the first metric, a second weight is applied to the second metric, and at least one of the first weight or the second weight is user-configurable;

adjust a confidence score of a prediction produced by the AI or ML model based, at least in part, on the weighted average of the first and second metrics;

modify the prediction based, at least in part, upon the identified drift, wherein modify the prediction further comprises individually adjust each inference of the plurality of inferences based, at least in part, on a location of a respective sensor device of the respective plurality of sensor devices; and

in response to a determination that drift exceeds a threshold, indicate an associated prediction is not trusted and initiate a pipeline that comprises corrective retrain, test, and deployment of a revised model based at least in part on a selected subset of tagged input data.

16 . The hardware memory device of claim 15 , wherein to adjust the confidence score, the program instructions, upon execution, further cause the IHS to, in response to a determination that the drift is greater than a first threshold value, reduce the confidence score.

17 . The hardware memory device of claim 16 , wherein to adjust the confidence score, the program instructions, upon execution, further cause the IHS to, in response to a determination that the drift is smaller than a second threshold value, increase the confidence score.

18 . A method, comprising:

determining a plurality of inferences produced by a respective plurality of trained Artificial Intelligence (AI) or Machine Learning (ML) models based on sensor data from a plurality of sensor devices;

identifying drift with respect to the plurality of trained AI or ML models, based, at least in part, on both a pre-model analysis of at least a variance of input data, and a post-model analysis of the plurality of inferences, wherein identifying drift further comprises:

providing at least a portion of the input data to a plurality of AI/ML drift detection models;

receiving respective drift confidence values from the plurality of AI/ML drift detection models;

selecting at least one AI/ML drift detection model having a smallest drift confidence value among the respective drift confidence values; and

tagging the input data with a characteristic label associated with the selected AI/ML drift detection model, wherein the characteristic comprises a hidden feature not detectable in the input data;

calculating a weighted average of a first metric from the pre-model analysis and a second metric from the post-model analysis, wherein a first weight is applied to the first metric, a second weight is applied to the second metric, and at least one of the first weight or the second weight is user-configurable;

adjusting a confidence score of a prediction produced by at least one AI or ML model based, at least in part, on the weighted average of the first and second metrics;

modifying the prediction based, at least in part, upon the identified drift, wherein modifying the prediction further comprises individually adjusting each inference of the plurality of inferences based, at least in part, on a location of a respective sensor device of the respective plurality of sensor devices; and

in response to determining that drift exceeds a threshold, indicating an associated prediction is not trusted and initiating a pipeline including corrective retraining, testing, and deployment of a revised plurality of AI or ML models using a selected subset of tagged input data.

19 . The method of claim 18 , wherein modifying the prediction further comprises indicating that the prediction is not trusted in response to the drift being greater than a first threshold value.

20 . The method of claim 19 , wherein modifying the prediction further comprises indicating that the prediction is trusted in response to the drift being smaller than a second threshold value.