Log anomaly detection in continuous artificial intelligence for it operations

1. A computer-implemented method for real-time statistical model based log anomaly detection, the method comprising:

receiving a windowed log of incoming raw log messages from a data source, the windowed log being as an inference input for log anomaly detection;

comparing statistical distribution metrics of entities in the windowed log with a statistical distribution extracted from a real-time statistical model for the entities, the entities being extracted from the incoming raw log messages and including message identifiers, log levels, error codes, and exception types;

in response to determining the statistical distribution metrics of the entities being statistically different from the statistical distribution extracted from the real-time statistical model for the entities, tagging the windowed log as an entity anomaly;

computing a distance between an average word embedding vector in the windowed log and a statistical distribution extracted form a real-time statistical model for word embeddings;

in response to determining the distance being greater than a predetermined threshold of the distance, tagging the windowed log as a word embedding anomaly; and

sending to a user an alert with an anomaly severity level.

2. The computer-implemented method of claim 1 , further comprising:

ensembling results of detecting the entity anomaly and detecting the word embedding anomaly;

in response to determining the windowed log being tagged as either the entity anomaly or the word embedding anomaly, sending the alert with a lower severity level; and

in response to determining the windowed log being tagged as both the entity anomaly and the word embedding anomaly, sending the alert with a higher severity level.

3. The computer-implemented method of claim 1 , further comprising:

in response to determining the statistical distribution metrics of the entities being not statistically different from the statistical distribution extracted from the real-time statistical model for the entities, tagging the windowed log as normal; and

in response to determining the distance being not greater than the predetermined threshold of the distance, tagging the windowed log as normal.

4. The computer-implemented method of claim 1 , further comprising:

querying the incoming raw log messages from the data source;

performing log entity extraction to obtain the entities in the incoming raw log messages;

generating the word embedding vectors for respective ones of the incoming raw log messages;

performing windowing for batches of the incoming raw log messages to generate windowed logs for respective ones of the batches of the incoming raw log messages;

aggregating the entities in each of the windowed logs, such that respective ones of the entities have entity names and entity counts in each of the windowed logs; and

averaging the word embedding vectors in each of the windowed logs, to obtain the average word embedding vector in each of the windowed logs.

5. The computer-implemented method of claim 1 , further comprising:

computing a ratio of a count of windowed logs tagged as anomalies to a total count of windowed logs in a previous time period just prior to a current time period of the log anomaly detection;

determining whether the ratio is greater than a predetermined threshold of the ratio; and

in response to determining the ratio being greater than the predetermined threshold of the ratio, skipping updating the real-time statistical model for the entities and the real-time statistical model for the word embeddings.

6. The computer-implemented method of claim 5 , further comprising:

in response to determining the ratio being not greater than the predetermined threshold of the ratio, retrieving the windowed logs in the previous time period;

accumulatively computing statistical distribution metrics of the entities in the windowed logs in the previous time period; and

updating the real-time statistical model for the entities, based on the statistical distribution metrics of the entities in the windowed logs in the previous time period.

7. The computer-implemented method of claim 6 , further comprising:

accumulatively computing statistical distribution metrics of the word embedding vectors in the windowed logs in the previous time period; and

updating the real-time statistical model for the word embeddings, based on the statistical distribution metrics of the word embedding vectors in the windowed logs in the previous time period.

8. A computer program product for real-time statistical model based log anomaly detection, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors, the program instructions executable to:

receive a windowed log of incoming raw log messages from a data source, the windowed log being as an inference input for log anomaly detection;

compare statistical distribution metrics of entities in the windowed log with a statistical distribution extracted from a real-time statistical model for the entities, the entities being extracted from the incoming raw log messages and including message identifiers, log levels, error codes, and exception types;

in response to determining the statistical distribution metrics of the entities being statistically different from the statistical distribution extracted from the real-time statistical model for the entities, tag the windowed log as an entity anomaly;

compute a distance between an average word embedding vector in the windowed log and a statistical distribution extracted form a real-time statistical model for word embeddings;

in response to determining the distance being greater than a predetermined threshold of the distance, tag the windowed log as a word embedding anomaly; and

send to a user an alert with an anomaly severity level.

9. The computer program product of claim 8 , further comprising the program instructions executable to:

ensemble results of detecting the entity anomaly and detecting the word embedding anomaly;

in response to determining the windowed log being tagged as either the entity anomaly or the word embedding anomaly, send the alert with a lower severity level; and

in response to determining the windowed log being tagged as both the entity anomaly and the word embedding anomaly, send the alert with a higher severity level.

10. The computer program product of claim 8 , further comprising the program instructions executable to:

in response to determining the statistical distribution metrics of the entities being not statistically different from the real-time statistical model for the entities, tag the windowed log as normal; and

in response to determining the distance being not greater than the predetermined threshold of the distance, tag the windowed log as normal.

11. The computer program product of claim 8 , further comprising the program instructions executable to:

query the incoming raw log messages from the data source;

perform log entity extraction to obtain the entities in the incoming raw log messages;

generate the word embedding vectors for respective ones of the incoming raw log messages;

perform windowing for batches of the incoming raw log messages to generate windowed logs for respective ones of the batches of the incoming raw log messages;

aggregate the entities in each of the windowed logs, such that respective ones of the entities have entity names and entity counts in each of the windowed logs; and

average the word embedding vectors in each of the windowed logs, to obtain the average word embedding vector in each of the windowed logs.

12. The computer program product of claim 8 , further comprising the program instructions executable to:

compute a ratio of a count of windowed logs tagged as anomalies to a total count of windowed logs in a previous time period just prior to a current time period of the log anomaly detection;

determine whether the ratio is greater than a predetermined threshold of the ratio; and

in response to determining the ratio being not greater than the predetermined threshold of the ratio, skip updating the real-time statistical model for the entities and the real-time statistical model for the word embeddings.

13. The computer program product of claim 12 , further comprising program instructions executable to:

in response to determining the ratio being greater than the predetermined threshold of the ratio, retrieve the windowed logs in the previous time period;

accumulatively compute statistical distribution metrics of the entities in the windowed logs in the previous time period; and

update the real-time statistical model for the entities, based on the statistical distribution metrics of the entities in the windowed logs in the previous time period.

14. The computer program product of claim 13 , further comprising the program instructions executable to:

accumulatively compute statistical distribution metrics of the word embedding vectors in the windowed logs in the previous time period; and

update the real-time statistical model for the word embeddings, based on the statistical distribution metrics of the word embedding vectors in the windowed logs in the previous time period.

15. A computer system for real-time statistical model based log anomaly detection, the computer system comprising one or more processors, one or more computer readable tangible storage devices, and program instructions stored on at least one of the one or more computer readable tangible storage devices for execution by at least one of the one or more processors, the program instructions executable to:

receive a windowed log of incoming raw log messages from a data source, the windowed log being as an inference input for log anomaly detection;

compare statistical distribution metrics of entities in the windowed log with a statistical distribution extracted from a real-time statistical model for the entities, the entities being extracted from the incoming raw log messages and including message identifiers, log levels, error codes, and exception types;

in response to determining the statistical distribution metrics of the entities being statistically different from the statistical distribution extracted from the real-time statistical model for the entities, tag the windowed log as an entity anomaly;

compute a distance between an average word embedding vector in the windowed log and a statistical distribution extracted form a real-time statistical model for word embeddings;

in response to determining the distance being greater than a predetermined threshold of the distance, tag the windowed log as a word embedding anomaly; and

send to a user an alert with an anomaly severity level.

16. The computer system of claim 15 , further comprising the program instructions executable to:

in response to determining the windowed log being tagged as either the entity anomaly or the word embedding anomaly, send the alert with a lower severity level; and

in response to determining the windowed log being tagged as both the entity anomaly and the word embedding anomaly, send the alert with a higher severity level.

17. The computer system of claim 15 , further comprising the program instructions executable to:

ensemble results of detecting the entity anomaly and detecting the word embedding anomaly;

in response to determining the statistical distribution metrics of the entities being not statistically different from the real-time statistical model for the entities, tag the windowed log as normal; and

in response to determining the distance being not greater than the predetermined threshold of the distance, tag the windowed log as normal.

18. The computer system of claim 15 , further comprising the program instructions executable to:

query the incoming raw log messages from the data source;

perform log entity extraction to obtain the entities in the incoming raw log messages;

generate the word embedding vectors for respective ones of the incoming raw log messages;

perform windowing for batches of the incoming raw log messages to generate windowed logs for respective ones of the batches of the incoming raw log messages;

aggregate the entities in each of the windowed logs, such that respective ones of the entities have entity names and entity counts in each of the windowed logs; and

average the word embedding vectors in each of the windowed logs, to obtain the average word embedding vector in each of the windowed logs.

19. The computer system of claim 15 , further comprising the program instructions executable to:

compute a ratio of a count of windowed logs tagged as anomalies to a total count of windowed logs in a previous time period just prior to a current time period of the log anomaly detection;

determine whether the ratio is greater than a predetermined threshold of the ratio; and

in response to determining the ratio being not greater than the predetermined threshold of the ratio, skip updating the real-time statistical model for the entities and the real-time statistical model for the word embeddings.

20. The computer system of claim 19 , further comprising program instructions executable to:

in response to determining the ratio being greater than the predetermined threshold of the ratio, retrieve the windowed logs in the previous time period;

accumulatively compute statistical distribution metrics of the entities in the windowed logs in the previous time period;

update the real-time statistical model for the entities, based on the statistical distribution metrics of the entities in the windowed logs in the previous time period;

accumulatively compute statistical distribution metrics of the word embedding vectors in the windowed logs in the previous time period; and

update the real-time statistical model for the word embeddings, based on the statistical distribution metrics of the word embedding vectors in the windowed logs in the previous time period.