Automated scoring of unstructured events in information technology environments
View Patent ↗Detection of anomalous events in the operation of information technology (IT) components includes receiving messages, which describe events in the operation of the IT components in real time, and categorizing and condensing the messages received in a first time interval into message patterns by message pattern type. Based on a distribution of occurrences of the message patterns in the first time interval and in preceding time intervals, anomaly scores are assigned to the message patterns, and one or more of the message patterns are classified as being anomalous message patterns that correspond to potentially anomalous events in the operation of the IT infrastructure installation. A degree of correlation between occurrences of the anomalous message patterns and occurrences of application alarms is determined. Message patterns with high anomaly scores and having a high degree of correlation with application alarms are deemed significant and prioritized for display to users.
1. A computer system for detecting anomalous events in the operation of information technology (IT) components of an IT infrastructure and prioritizing the anomalous events for remediation in the IT infrastructure, the computer system comprising:
a processor coupled to a memory;
a database receiving messages that in real time describe events in the operation of the IT components, and storing the received messages as raw messages;
a signature generator categorizing the raw messages received in a first-time interval by message pattern type and condensing the categorized raw messages into message patterns;
an anomaly scorer classifying a message pattern as being an anomalous message pattern that corresponds to an anomalous event in the operation of the IT components; and
an application mapper that:
assigns a significance value to the anomalous message pattern based on an assessment of an impact of the anomalous event on an availability or a performance of an application in the IT infrastructure; and
displays the anomalous message pattern based on the significance value assigned to the anomalous message pattern.
2. The computer system of claim 1 , wherein the signature generator is configured to replace varying content in the categorized raw messages with placeholders to form the message patterns.
3. The computer system of claim 1 , wherein the signature generator, when condensing the categorized raw messages into the message patterns, is configured to:
determine which of the categorized raw messages are similar to which other raw messages received in the first-time interval based on a string distance threshold between contents of the raw messages;
aggregate the categorized raw messages into one or more message groups of the similar raw messages received in the first-time interval; and
replace varying content in the similar raw messages in each of the one or more message groups by a respective placeholder to form a respective message pattern for each of the one or more message groups of the similar raw messages received in the first-time interval.
4. The computer system of claim 3 , wherein the signature generator is configured to:
compare the respective message pattern for each of the one or more message groups of the similar raw messages received in the first-time interval with a master message pattern previously stored in the database; and,
when the respective message pattern and the previously stored master message pattern are similar based on a string distance threshold, merge the respective message pattern and the previously stored master message pattern.
5. The computer system of claim 1 , wherein an anomaly scoring component is configured to assign a respective anomaly score to the message pattern based on a Median Absolute Divergence (MAD) parameter of a distribution of occurrences of the message patterns in an anomaly scoring time period.
6. The computer system of claim 5 , wherein the anomaly scoring component is configured to assign a maximum or a near maximum anomaly score to the message pattern that is an outlier relative to a median of an absolute divergence from the median of a distribution of occurrences of the message patterns in the anomaly scoring time period.
7. The computer system of claim 1 , wherein the signature generator is configured to:
keep count of occurrences of the anomalous message pattern in the first-time interval; and,
store the count along with counts of previous occurrences of the anomalous message pattern in the one or more preceding time intervals.
8. The computer system of claim 1 , wherein the application mapper is configured to:
determine a degree of correlation between occurrences of the anomalous message pattern and occurrences of an application alarm of the application over a correlation time period including the first-time interval; and
assign the significance value to the anomalous message pattern based on the determined degree of correlation.
9. A method for detecting anomalous events in the operation of information technology (IT) components of an IT infrastructure and prioritizing the anomalous events for remediation in the IT infrastructure, the method comprising:
receiving messages that describe events in the operation of the IT components in real time and storing the received messages as raw messages in a database;
categorizing the raw messages received in a first-time interval by message pattern type and condensing the categorized raw messages into message patterns;
classifying a message pattern in the first-time interval as being an anomalous message pattern that corresponds to an anomalous event in the operation of the IT components;
assigning a significance value to the anomalous message pattern based on an assessment of an impact of the anomalous event on an availability or a performance of an application in the IT infrastructure; and
displaying the anomalous message pattern based on the significance value assigned to the anomalous message pattern.
10. The method of claim 9 , wherein condensing the categorized raw messages in to message patterns includes replacing varying content in the categorized raw messages by placeholders to form the message patterns.
11. The method of claim 9 , wherein condensing the categorized raw messages in to message patterns includes:
determining which of the categorized raw messages are similar to which other raw messages received in the first-time interval based on a string distance threshold between contents of the raw messages;
aggregating the categorized raw messages into one or more message groups of the similar raw messages received in the first time interval; and
replacing varying content in the similar raw messages in each of the one or more message groups by a respective placeholder to form a respective message pattern for each of the one or more message groups of the similar raw messages received in the first-time interval.
12. The method of claim 11 , wherein condensing the categorized raw messages into message patterns includes:
comparing the respective message pattern for each of the one or more message groups of the similar raw messages received in the first-time interval with a master message pattern previously stored in the database; and
when the respective message pattern and the previously stored master message pattern are similar based on a string distance threshold, merging the respective message pattern and the previously stored master message pattern.
13. The method of claim 9 , wherein classifying the message pattern in the first-time interval as being an anomalous message pattern includes assigning a respective anomaly score to the message pattern based on a Median Absolute Divergence (MAD) parameter of a distribution of occurrences of the message patterns in an anomaly scoring time period.
14. The method of claim 13 , wherein assigning a respective anomaly score to each of the message patterns includes assigning a maximum or a near maximum anomaly score to a message pattern that is an outlier relative to a median of an absolute divergence from the median of a distribution of occurrences of the message patterns in the anomaly scoring time period.
15. The method of claim 9 further comprising:
keeping count of occurrences of the anomalous message pattern in the first-time interval; and,
storing the count along with counts of previous occurrences of the anomalous message pattern in the one or more preceding time intervals.
16. The method of claim 9 , wherein assigning a significance value to the anomalous message pattern includes:
determining a degree of correlation between occurrences of the anomalous message pattern and occurrences of an application alarm of the application over a correlation time period including the first-time interval; and,
assigning the significance value to the anomalous message pattern based on the determined degree of correlation.
17. A computer program product for detecting anomalous events in the operation of information technology (IT) components of an IT infrastructure and prioritizing the anomalous events for remediation in the IT infrastructure, the computer program product including instructions recorded on a non-transitory computer-readable storage medium and configured to cause at least one processor to:
receive messages that in real time describe events in the operation of IT components and store the received messages as raw messages in a database;
categorize the raw messages received in a first-time interval by message pattern type and condense the categorized raw messages into message patterns;
classify a message pattern in the first-time interval as being an anomalous message pattern that corresponds to an anomalous event in the operation of the IT components;
assign a significance value to the anomalous message pattern based on an assessment of an impact of the anomalous event on an availability or a performance of an application in the IT infrastructure; and
display the anomalous message pattern based on the significance value assigned to the anomalous message pattern.
18. The computer program product of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
replace varying content in the categorized raw messages with placeholders to form the message patterns.
19. The computer program product of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor, when condensing the categorized raw messages into message patterns, to:
determine which of the categorized raw messages are similar to which other raw messages received in the first-time interval based on a string distance threshold between the contents of the raw messages;
aggregate the categorized raw messages into one or more message groups of the similar raw messages received in the first-time interval; and
replace the varying content in the similar raw messages in each of the one or more message groups by a respective placeholder to form a respective message pattern for each of the one or more message groups of the similar raw messages received in the first-time interval.
20. The computer program product of claim 19 , wherein the instructions, when executed, are further configured to cause the at least one processor, when condensing the categorized raw messages into message patterns, to:
compare the respective message pattern for each of the one or more message groups of the similar raw messages received in the first-time interval with a master message pattern previously stored in the database; and
when the respective message pattern and the previously stored master message pattern are similar based on a string distance threshold, merge the respective message pattern and the previously stored master message pattern.
21. The computer program product of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor, when classifying the message pattern in the first-time interval as being an anomalous message pattern, to:
assign a respective anomaly score to the message pattern based on a Median Absolute Divergence (MAD) parameter of the distribution of occurrences of the message patterns in an anomaly scoring time period.
22. The computer program product of claim 21 , wherein the instructions, when executed, are further configured to cause the at least one processor to assign a maximum or a near maximum anomaly score to a message pattern that is an outlier relative to a median of an absolute divergence from the median of a distribution of occurrences of the message patterns in the anomaly scoring time period.
23. The computer program product of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
keep count of occurrences of the anomalous message pattern in the first-time interval; and,
store the count along with counts of previous occurrences of the anomalous message pattern in the one or more preceding time intervals.
24. The computer program product of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
determine a degree of correlation between occurrences of the anomalous message pattern and occurrences of an application alarm of the application over a correlation time period including the first-time interval; and
assign the significance value to the anomalous message pattern based on the determined degree of correlation.