Short-term model calibration in system monitoring
A technology landscape may be characterized using a performance characterization that includes scores assigned to performance metrics for the technology landscape and using at least one trained machine learning model. In response to a detected calibration trigger, a calibratable performance metric of the performance metrics may be determined. A relationship may be determined between conforming values of the calibratable performance metric during a conforming period for which the at least one trained machine learning model was trained, and non-conforming values of the calibratable performance metric occurring during a calibration period initiated by the calibration trigger. In this way, a score assigned to the calibratable performance metric may be calibrated, based on the relationship.
1. A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:
detect a calibration trigger for a technology landscape, the technology landscape being characterized using a performance characterization that includes scores assigned to performance metrics for the technology landscape and using at least one trained machine learning model;
determine, in response to the calibration trigger, a calibratable performance metric of the performance metrics;
determine a relationship between conforming values of the calibratable performance metric during a conforming period for which the at least one trained machine learning model was trained, and non-conforming values of the calibratable performance metric occurring during a calibration period initiated by the calibration trigger; and
calibrate a score assigned to the calibratable performance metric by the at least one trained machine learning model during the calibration period, based on the relationship, to thereby obtain a calibrated score for inclusion in the performance characterization.
2. The computer program product of claim 1 , wherein the calibration trigger includes a detection of occurrence of a calibration date.
3. The computer program product of claim 1 , wherein the calibration trigger includes a detection of a violation of a false positive threshold of the scores.
4. The computer program product of claim 1 , wherein the relationship includes a ratio of the non-conforming values and the conforming values.
5. The computer program product of claim 4 , wherein the ratio is calculated using non-conforming values and conforming values occurring during similar times of day as one another during the calibration period and the conforming period, respectively.
6. The computer program product of claim 1 , wherein the calibratable performance metric includes a workload metric associated with program logic of an application running in the technology landscape.
7. The computer program product of claim 1 , wherein the instructions, when executed to calibrate the score, are further configured to cause the at least one computing device to:
adjust a non-conforming value from among the non-conforming values to obtain an adjusted value; and
provide the adjusted value to the at least one trained machine learning model to obtain the calibrated score.
8. The computer program product of claim 1 , wherein the instructions, when executed to calibrate the score, are further configured to cause the at least one computing device to:
determine a shifted scoring threshold for use during the calibration period, the shifted scoring threshold being related to a conforming scoring threshold used during the conforming period by a shifted amount;
score a non-conforming value from among the non-conforming values using the at least one trained machine learning model to obtain an uncalibrated score; and
shift the uncalibrated score by the shifted amount to obtain the calibrated score.
9. The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
detect a subsequent calibration trigger that extends the calibration period;
following the subsequent calibration trigger and applicable within the extended calibration period, detect an updated relationship between the conforming values and the non-conforming values; and
calibrate an updated score assigned to the calibratable performance metric by the at least one trained machine learning model during the extended calibration period to thereby obtain an updated calibrated score for inclusion in the performance characterization.
10. The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
update the relationship during the calibration period and with a minimum update frequency; and
update the calibration of scoring the calibratable performance metric value with each update of the relationship.
11. A computer-implemented method, the method comprising:
detecting a calibration trigger for a technology landscape, the technology landscape being characterized using a performance characterization that includes scores assigned to performance metrics for the technology landscape and using at least one trained machine learning model;
determining, in response to the calibration trigger, a calibratable performance metric of the performance metrics;
determining a relationship between conforming values of the calibratable performance metric during a conforming period for which the at least one trained machine learning model was trained, and non-conforming values of the calibratable performance metric occurring during a calibration period initiated by the calibration trigger; and
calibrating a score assigned to the calibratable performance metric by the at least one trained machine learning model during the calibration period, based on the relationship, to thereby obtain a calibrated score for inclusion in the performance characterization.
12. The method of claim 11 , wherein the calibration trigger includes a detection of occurrence of a calibration date.
13. The method of claim 11 , wherein the calibration trigger includes a detection of a violation of a false positive threshold of the scores.
14. The method of claim 11 , wherein the relationship includes a ratio of the non-conforming values and the conforming values.
15. The method of claim 14 , wherein the ratio is calculated using non-conforming values and conforming values occurring during similar times of day as one another during the calibration period and the conforming period, respectively.
16. The method of claim 11 , further comprising:
detecting a subsequent calibration trigger that extends the calibration period;
following the subsequent calibration trigger and applicable within the extended calibration period, detecting an updated relationship between the conforming values and the non-conforming values; and
calibrating an updated score assigned to the calibratable performance metric by the at least one trained machine learning model during the extended calibration period, to thereby obtain an updated calibrated score for inclusion in the performance characterization.
17. The method of claim 11 , further comprising:
updating the relationship during the calibration period and with a minimum update frequency; and
updating the calibration of scoring the calibratable performance metric value with each update of the relationship.
18. A system comprising:
at least one memory including instructions; and
at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the at least one processor to
detect a calibration trigger for a technology landscape, the technology landscape being characterized using a performance characterization that includes scores assigned to performance metrics for the technology landscape and using at least one trained machine learning model;
determine, in response to the calibration trigger, a calibratable performance metric of the performance metrics;
determine a relationship between conforming values of the calibratable performance metric during a conforming period for which the at least one trained machine learning model was trained, and non-conforming values of the calibratable performance metric occurring during a calibration period initiated by the calibration trigger; and
calibrate a score assigned to the calibratable performance metric by the at least one trained machine learning model during the calibration period, based on the relationship, to thereby obtain a calibrated score for inclusion in the performance characterization.
19. The system of claim 18 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
detect a subsequent calibration trigger that extends the calibration period;
following the subsequent calibration trigger and applicable within the extended calibration period, detect an updated relationship between the conforming values and the non-conforming values; and
calibrate an updated score assigned to the calibratable performance metric by the at least one trained machine learning model during the extended calibration period, to thereby obtain an updated calibrated score for inclusion in the performance characterization.
20. The system of claim 18 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
update the relationship during the calibration period and with a minimum update frequency; and
update the calibration of scoring the calibratable performance metric value with each update of the relationship.