Framework for measuring telemetry data variability for confidence evaluation of a machine learning estimator
A deployment manager includes storage for storing a prediction model based on telemetry data from the deployments and a prediction manager. The prediction manager generates, using the prediction model and second telemetry data obtained from a deployment of the deployments: a prediction, and a prediction error estimate; in response to a determination that the prediction indicates a negative impact on the deployment: generates a confidence estimation for the prediction based on a variability of the second telemetry data from the telemetry data; in response to a second determination that the confidence estimation indicates that the prediction error estimate is inaccurate: remediates the prediction based on the variability to obtain an updated prediction; and performs an action set, based on the updated prediction, to reduce an impact of the negative impact on the deployment.
1. A deployment manager, comprising:
storage for storing a prediction model based on telemetry data from deployments; and
a prediction manager programmed to:
generate, using the prediction model and second telemetry data obtained from a deployment of the deployments:
a prediction, and
a prediction error estimate;
in response to a determination that the prediction indicates a negative impact on the deployment:
generate a confidence estimation for the prediction based on a variability of the second telemetry data from the telemetry data;
in response to a second determination that the confidence estimation indicates that the prediction error estimate is inaccurate:
remediate the prediction based on the variability to obtain an updated prediction; and
perform an action set, based on the updated prediction, to reduce an impact of the negative impact on the deployment.
2. The deployment manager of claim 1 , wherein the prediction manager is further programmed to:
in response to a third determination that a second prediction indicates a second negative impact on a second deployment of the deployments:
generate a second confidence estimation for the second prediction based on a variability of third telemetry data, from the second deployment, from the telemetry data;
in response to a fourth determination that a second confidence estimation, associated with the second prediction, indicates that the second prediction error estimate is inaccurate:
perform a second action set, based on the second prediction, to reduce an impact of the second negative impact on the second deployment.
3. The deployment manager of claim 1 , wherein generating the confidence estimation comprises:
obtaining probability distributions for portions of the telemetry data associated with each deployment of the deployments.
4. The deployment manager of claim 3 , wherein generating the confidence estimation further comprises:
obtaining, using the probability distributions, the variability.
5. The deployment manager of claim 4 , wherein generating the confidence estimation further comprises:
obtaining a correlation of the variability with the prediction error estimate.
6. The deployment manager of claim 5 , wherein the confidence estimation is based on the correlation.
7. The deployment manager of claim 1 , wherein the action set comprises:
modifying an operation of the deployment based on the prediction.
8. A method for generating predictions using a prediction model based on telemetry data from deployments, comprising:
generating, using the prediction model and second telemetry data obtained from a deployment of the deployments:
a prediction, and
a prediction error estimate;
in response to a determination that the prediction indicates a negative impact on the deployment:
generating a confidence estimation for the prediction based on a variability of the second telemetry data from the telemetry data;
in response to a second determination that the confidence estimation indicates that the prediction error estimate is inaccurate:
remediating the prediction based on the variability to obtain an updated prediction; and
performing an action set, based on the updated prediction, to reduce an impact of the negative impact on the deployment.
9. The method of claim 8 , further comprising:
in response to a third determination that a second prediction indicates a second negative impact on a second deployment of the deployments:
generating a second confidence estimation for the second prediction based on a variability of third telemetry data, from the second deployment, from the telemetry data;
in response to a fourth determination that a second confidence estimation, associated with the second prediction, indicates that the second prediction error estimate is inaccurate:
performing a second action set, based on the second prediction, to reduce an impact of the second negative impact on the second deployment.
10. The method of claim 8 , wherein generating the confidence estimation comprises:
obtaining probability distributions for portions of the telemetry data associated with each deployment of the deployments.
11. The method of claim 10 , wherein generating the confidence estimation further comprises:
obtaining, using the probability distributions, the variability.
12. The method of claim 11 , wherein generating the confidence estimation further comprises:
obtaining a correlation of the variability with the prediction error estimate.
13. The method of claim 12 , wherein the confidence estimation is based on the correlation.
14. The method of claim 8 , wherein the action set comprises:
modifying an operation of the deployment based on the prediction.
15. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for generating predictions using a prediction model based on telemetry data from deployments, the method comprising:
generating, using the prediction model and second telemetry data obtained from a deployment of the deployments:
a prediction, and
a prediction error estimate;
in response to a determination that the prediction indicates a negative impact on the deployment:
generating a confidence estimation for the prediction based on a variability of the second telemetry data from the telemetry data;
in response to a second determination that the confidence estimation indicates that the prediction error estimate is inaccurate:
remediating the prediction based on the variability to obtain an updated prediction; and
performing an action set, based on the updated prediction, to reduce an impact of the negative impact on the deployment.
16. The non-transitory computer readable medium of claim 15 , wherein the method further comprises:
in response to a third determination that a second prediction indicates a second negative impact on a second deployment of the deployments:
generating a second confidence estimation for the second prediction based on a variability of third telemetry data, from the second deployment, from the telemetry data;
in response to a fourth determination that a second confidence estimation, associated with the second prediction, indicates that the second prediction error estimate is inaccurate:
performing a second action set, based on the second prediction, to reduce an impact of the second negative impact on the second deployment.
17. The non-transitory computer readable medium of claim 15 , wherein generating the confidence estimation comprises:
obtaining probability distributions for portions of the telemetry data associated with each deployment of the deployments.
18. The non-transitory computer readable medium of claim 17 , wherein generating the confidence estimation further comprises:
obtaining, using the probability distributions, the variability.
19. The non-transitory computer readable medium of claim 17 , wherein generating the confidence estimation further comprises:
obtaining a correlation of the variability with the prediction error estimate.
20. The non-transitory computer readable medium of claim 19 , wherein the confidence estimation is based on the correlation.