IP Library › Granted Patent US 12,309,173
Granted Patent B2
US 12,309,173 · App. 17/344,751 · Granted May 20, 2025

Performance monitoring in the anomaly detection domain for the IT environment

Inventors: Shirley M. Han (New York, NY); Anbang Xu (San Jose, CA); Rama Kalyani T. Akkiraju (Cupertino, CA); Salil Ahuja (Washington, DC); Xiaotong Liu (San Jose, CA)
Assignee: International Business Machines Corporation
H04L63/1416G06N5/04G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,309,173
App. No.
17/344,751
Granted
May 20, 2025
Kind
B2
Abstract

An approach is disclosed that retrieves a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model that is monitoring the computer system. The current system data and the anomaly detection model outputs are input to a trained anomaly detection supervisor model. The trained anomaly detection supervisor model processes the inputs and provides a set of performance data corresponding to the anomaly detection model. The anomaly detection model is then adjusted when the set of performance data indicates that the anomaly detection model is performing below a threshold.

Claims (67)

1. A computer-implemented method, implemented by an information handling system that includes a processor and a memory, the method comprising:

outputting a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model that is monitoring the computer system;

inputting the set of current system data and the set of current outputs to a trained anomaly detection supervisor model including inputting a set of current system log templates to the trained anomaly detection supervisor model that compares the set of current system log templates to one or more sets of older system log templates, wherein the comparison results in one of performance data that indicates whether the anomaly detection model is performing below a threshold;

receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model, wherein the received performance data is based on the set of current system data and the set of current outputs; and

adjusting the anomaly detection model in response to the set of performance data indicating that the anomaly detection model is performing below the threshold.

2. The method of claim 1 further comprising:

training an anomaly detection supervisor model, wherein the training results in the trained anomaly detection supervisor model, and wherein the training further comprises:

inputting a training inference dataset to the anomaly detection supervisor model, wherein the training inference dataset is an output of the anomaly detection model that is monitoring the computer system; and

wherein the adjusting of the anomaly detection model is selected from the group consisting of retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

3. The method of claim 2 further comprising:

receiving a current inference dataset that is output from the anomaly detection model that is monitoring the computer system; and

using the trained anomaly detection supervisor model to compare the current inference dataset with one or more older inference datasets, wherein one of the older inference datasets is the training inference dataset, wherein the comparing results in one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

4. The method of claim 1 further comprising:

training an anomaly detection supervisor model, wherein the training results in the trained anomaly detection supervisor model, and wherein the training further comprises:

inputting a training reference count vector corresponding to a predetermined timeframe to the anomaly detection supervisor model, wherein the training reference count vector is an output of the anomaly detection model that is monitoring the computer system.

5. The method of claim 4 further comprising:

receiving a reference count vector that is output from the anomaly detection model that is monitoring the computer system; and

using the trained anomaly detection supervisor model to compare the current reference count vector with one or more older reference count vectors, wherein one of the older reference count vectors is the training reference count vector, wherein the comparing results in one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

6. The method of claim 1 further comprising:

inputting a set of current system metric data to the trained anomaly detection supervisor model that compares the set of current system metric data to one or more sets of older system metric data, wherein the comparison results in an analysis of data drift in the anomaly detection model, and wherein the analysis of the data drift is one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

7. The method of claim 1 further comprising:

inputting a set of current system metric data to the trained anomaly detection supervisor model that compares the set of current system metric data to one or more sets of older system metric data, wherein the comparison results in an analysis of data drift in the anomaly detection model, and wherein the analysis of the data drift is one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

8. An information handling system comprising:

one or more processors;

a memory coupled to at least one of the processors;

a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions comprising:

outputting a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model that is monitoring the computer system;

inputting the set of current system data and the set of current outputs to a trained anomaly detection supervisor model including inputting a set of current system log templates to the trained anomaly detection supervisor model that compares the set of current system log templates to one or more sets of older system log templates, wherein the comparison results in one of performance data that indicates whether the anomaly detection model is performing below a threshold:

receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model, wherein the received performance data is based on the set of current system data and the set of current outputs; and

adjusting the anomaly detection model in response to the set of performance data indicating that the anomaly detection model is performing below the threshold.

9. The information handling system of claim 8 wherein the actions further comprise:

training an anomaly detection supervisor model, wherein the training results in the trained anomaly detection supervisor model, and wherein the training further comprises:

inputting a training inference dataset to the anomaly detection supervisor model, wherein the training inference dataset is an output of the anomaly detection model that is monitoring the computer system; and

wherein the adjusting of the anomaly detection model is selected from the group consisting of retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

10. The information handling system of claim 9 wherein the actions further comprise:

receiving a current inference dataset that is output from the anomaly detection model that is monitoring the computer system; and

using the trained anomaly detection supervisor model to compare the current inference dataset with one or more older inference datasets, wherein one of the older inference datasets is the training inference dataset, wherein the comparing results in one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

11. The information handling system of claim 8 wherein the actions further comprise:

training an anomaly detection supervisor model, wherein the training results in the trained anomaly detection supervisor model, and wherein the training further comprises:

inputting a training reference count vector corresponding to a predetermined timeframe to the anomaly detection supervisor model, wherein the training reference count vector is an output of the anomaly detection model that is monitoring the computer system.

12. The information handling system of claim 11 wherein the actions further comprise:

receiving a reference count vector that is output from the anomaly detection model that is monitoring the computer system; and

using the trained anomaly detection supervisor model to compare the current reference count vector with one or more older reference count vectors, wherein one of the older reference count vectors is the training reference count vector, wherein the comparing results in one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

13. The information handling system of claim 8 wherein the actions further comprise:

inputting a set of current system metric data to the trained anomaly detection supervisor model that compares the set of current system metric data to one or more sets of older system metric data, wherein the comparison results in an analysis of data drift in the anomaly detection model, and wherein the analysis of the data drift is one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

14. The information handling system of claim 8 wherein the actions further comprise:

inputting a set of current system metric data to the trained anomaly detection supervisor model that compares the set of current system metric data to one or more sets of older system metric data, wherein the comparison results in an analysis of data drift in the anomaly detection model, and wherein the analysis of the data drift is one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

15. A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, performs actions comprising:

outputting a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model that is monitoring the computer system;

inputting the set of current system data and the set of current outputs to a trained anomaly detection supervisor model including inputting a set of current system log templates to the trained anomaly detection supervisor model that compares the set of current system log templates to one or more sets of older system log templates, wherein the comparison results in one of performance data that indicates whether the anomaly detection model is performing below a threshold:

receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model, wherein the received performance data is based on the set of current system data and the set of current outputs; and

adjusting the anomaly detection model in response to the set of performance data indicating that the anomaly detection model is performing below the threshold.

16. The computer program product of claim 15 wherein the actions further comprise:

training an anomaly detection supervisor model, wherein the training results in the trained anomaly detection supervisor model, and wherein the training further comprises:

inputting a training inference dataset to the anomaly detection supervisor model, wherein the training inference dataset is an output of the anomaly detection model that is monitoring the computer system; and

wherein the adjusting of the anomaly detection model is selected from the group consisting of retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

17. The computer program product of claim 16 wherein the actions further comprise:

receiving a current inference dataset that is output from the anomaly detection model that is monitoring the computer system; and

using the trained anomaly detection supervisor model to compare the current inference dataset with one or more older inference datasets, wherein one of the older inference datasets is the training inference dataset, wherein the comparing results in one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

18. The computer program product of claim 15 wherein the actions further comprise:

training an anomaly detection supervisor model, wherein the training results in the trained anomaly detection supervisor model, and wherein the training further comprises:

inputting a training reference count vector corresponding to a predetermined timeframe to the anomaly detection supervisor model, wherein the training reference count vector is an output of the anomaly detection model that is monitoring the computer system.

19. The computer program product of claim 18 wherein the actions further comprise:

receiving a reference count vector that is output from the anomaly detection model that is monitoring the computer system; and

using the trained anomaly detection supervisor model to compare the current reference count vector with one or more older reference count vectors, wherein one of the older reference count vectors is the training reference count vector, wherein the comparing results in one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

20. The computer program product of claim 15 wherein the actions further comprise:

inputting a set of current system metric data to the trained anomaly detection supervisor model that compares the set of current system metric data to one or more sets of older system metric data, wherein the comparison results in an analysis of data drift in the anomaly detection model, and wherein the analysis of the data drift is one of the performance data that indicates whether the anomaly detection model is performing below the threshold.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 10, 2021
From: HAN, SHIRLEY M.; XU, ANBANG; AKKIRAJU, RAMA KALYANI T.; AHUJA, SALIL; LIU, XIAOTONG
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 056505/0800 →
Continuity (1)
Related Publication 20220400121A1 · Dec 15, 2022
References Cited (11)
US 10599957B2 · Walters · 2020 [cited by applicant]
US 10970182B2 · Chafle · 2021 [cited by applicant]
US 20170372232A1 · Maughan · 2017 [cited by applicant]
US 20180096261A1 · Chu · 2018 [cited by applicant]
US 20180247220A1 · Assem Aly Salama · 2018 [cited by examiner]
US 20200076677A1 · Mermoud · 2020 [cited by examiner]
US 20210209486A1 · Fan · 2021 [cited by examiner]
CN 110956278A · 2020 [cited by applicant]
Anonymously, “Compressed Sensing for Monitoring of an Application,” IP.com No. IPCOM000252103D, Dec. 15, 2017, 33 pages. [cited by applicant]
Liu et al., “Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning,” Proceedings of the 2015 Internet Measurement Conference, pp. 1-14, Oct. 2015, New York, NY. [cited by applicant]
Yuan et al., “Ada: Adaptive Deep Log Anomaly Detector,” IEEE Infocom 2020—IEEE Conference on Computer Communications, pp. 2449-2458, Jul. 2020. [cited by applicant]