Predictive modeling to identify anomalous log data
Disclosed herein are methods, systems, and processes for interference-based detection and identification of anomalous log data using predictive modeling. A log data that includes a path with strings is accessed. Multiple anomalous log data prediction models are trained for the path by processing the strings at a character level and at a name level using disparate Markov prediction models that include n-gram and skip gram models after performing an A-replace operation. A trained dataset is generated based on the training that includes a simplified path for each of the various anomalous log data prediction models along with a transition probability for each string in the path. Other paths in the log or other logs are trained using the trained dataset and the several trained anomalous log data prediction models are deployed to observe, identify, and highlight anomalous strings in new log data.
1. A computer-implemented method, comprising:
accessing a log comprising a path with a plurality of strings;
pre-processing the log to generate an updated dataset, wherein the updated dataset includes a simplified path generated based on the path, wherein the simplified path is generated by performing an A-replace operation on the path, wherein the A-replace operation replaces each sequence of characters in a particular character set with a single-character sequence of a same length as the replaced sequence;
training a plurality of anomalous log data prediction models using the updated dataset by processing simplified paths in the updated dataset at a character level and at a name level, wherein the training includes building Markov prediction chains comprising transition probabilities between sequential elements of the simplified paths;
generating a character heatmap on a graphical user interface (GUI) for each simplified path processed by the plurality of anomalous log data prediction models based on the transition probability for each string of each path, wherein the transition probability indicates a probability of characters in each string of each path, and the character heatmap designates each character in each simplified path as high likelihood or low likelihood; and
displaying on the GUI one or more anomalies detected in the log, wherein the one or more anomalies are detected based on the plurality of anomalous log data prediction models and the character heatmap.
2. The computer-implemented method of claim 1 , wherein
the A-replace operation comprises performing regular expression (regex) character replacement for one or more special characters in the plurality of strings by replacing (a) one or more alphanumeric characters, (b) one or more numbers, or (c) one or more upper case characters, one or more low case characters, and one or more numbers, and
performing the regex character replacement identifies one or more common patterns of special characters that comprise globally unique identifiers (GUIDs).
3. The computer-implemented method of claim 1 , wherein the GUI identifies an anomaly in the log by displaying a sequence of characters in a path in a bolded, underlined, or bigger font, wherein the sequence of characters is determined to be unusual based on the anomalous log data prediction models.
4. The computer-implemented method of claim 1 , further comprising:
receiving a new process event;
extracting a new path from the new process event;
processing the new path using the plurality of trained anomalous log data prediction models that are trained using the updated dataset;
generating an updated character heatmap that indicates a likelihood of each character in the new path predicted by each of the plurality of trained anomalous log data prediction models; and
generating a regex replacement or normalization for (a) each of one or more new strings in the new path and (b) for one or more existing strings in the path or in other paths in the log that creates a new simplified path for one or more of the plurality of trained anomalous log data prediction models.
5. The computer-implemented method of claim 4 , further comprising:
accessing the updated character heatmap;
comparing each of the new simplified paths for each of the plurality of trained anomalous log data prediction models based on the likelihood that each character in the new path is high likelihood or low likelihood; and
based on the comparing, identifying one or more new simplified paths of one or more trained anomalous log data prediction models as anomalous.
6. The computer-implemented method of claim 1 , further comprising:
receiving historical log data;
batching the historical log data into a plurality of paths; and
filtering the plurality of paths based on a single type of string of one or more types of strings that are part of the plurality of paths.
7. The computer-implemented method of claim 6 , wherein
the one or more types of strings comprise a file path and a child process, and
the plurality of strings comprise a plurality of key/value (KV) pairs.
8. A non-transitory computer readable storage medium comprising program instructions executable to:
access a log comprising a path with a plurality of strings;
pre-process the log to generate an updated dataset, wherein the updated dataset includes a simplified path generated based on the path, wherein the simplified path is generated by performing an A-replace operation on the path, wherein the A-replace operation replaces each sequence of characters in a particular character set with a single-character sequence of a same length as the replaced sequence;
train a plurality of anomalous log data prediction models using the updated dataset by processing simplified paths in the updated dataset at a character level and at a name level to build Markov prediction chains comprising transition probabilities between sequential elements of the simplified paths;
generate a character heatmap on a graphical user interface (GUI) for each simplified path processed by the plurality of anomalous log data prediction models based on the transition probability for each string of each path, wherein the transition probability indicates a probability of characters in each string of each path, and the character heatmap designates each character in each simplified path as high likelihood or low likelihood; and
display on the GUI one or more anomalies detected in the log, wherein the one or more anomalies are detected based on use the plurality of anomalous log data prediction models to detect anomalies and the character heatmap.
9. The non-transitory computer readable storage medium of claim 8 , wherein
the A-replace operation comprises performing regular expression (regex) character replacement for one or more special characters in the plurality of strings by replacing (a) one or more alphanumeric characters, (b) one or more numbers, or (c) one or more upper case characters, one or more low case characters, and one or more numbers, and
performing the regex character replacement identifies one or more common patterns of special characters that comprise globally unique identifiers (GUIDs).
10. The non-transitory computer readable storage medium of claim 8 , wherein the program instructions are executable to:
cause the GUI to identify an anomaly in the log by displaying a sequence of characters in a path in a bolded, underlined, or bigger font, wherein the sequence of characters is determined to be unusual based on the anomalous log data prediction models.
11. The non-transitory computer readable storage medium of claim 8 , wherein the program instructions are executable to:
receive a new process event;
extract a new path from the new process event;
process the new path using the plurality of trained anomalous log data prediction models that are trained using the updated dataset that comprises a plurality of transition probabilities of a plurality of paths;
generate an updated character heatmap that indicates a likelihood of each character in the new path predicted by each of the plurality of trained anomalous log data prediction models; and
generate a regex replacement or normalization for (a) each of one or more new strings in the new path and (b) for one or more existing strings in the path or in other paths in the log that creates a new simplified path for one or more of the plurality of trained anomalous log data prediction models.
12. The non-transitory computer readable storage medium of claim 11 , wherein the program instructions are executable to:
access the updated character heatmap;
compare each of the new simplified paths for each of the plurality of trained anomalous log data prediction models based on the likelihood that each character in the new path is high likelihood or low likelihood; and
based on the comparing, identify one or more new simplified paths of one or more trained anomalous log data prediction models as anomalous.
13. The non-transitory computer readable storage medium of claim 8 , wherein the program instructions are executable to:
receive historical log data;
batch the historical log data into a plurality of paths; and
filter the plurality of paths based on a single type of string of one or more types of strings that are part of the plurality of paths.
14. The non-transitory computer readable storage medium of claim 13 , wherein
the one or more types of strings comprise a file path and a child process, and
the plurality of strings comprise a plurality of key/value (KV) pairs.
15. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to:
access a log comprising a path with a plurality of strings;
pre-process the log to generate an updated dataset, wherein the updated dataset includes a simplified path generated based on the path, wherein the simplified path is generated by performing an A-replace operation on the path, wherein the A-replace operation replaces each sequence of characters in a particular character set with a single-character sequence of a same length as the replaced sequence;
train a plurality of anomalous log data prediction models using the updated dataset by processing simplified paths in the updated dataset at a character level and at a name level to build Markov prediction chains comprising transition probabilities between sequential elements of the simplified paths;
generate a character heatmap on a graphical user interface (GUI) for each simplified path processed by the plurality of anomalous log data prediction models based on the transition probability for each string of each path, wherein the transition probability indicates a probability of characters in each string of each path, and the character heatmap designates each character in each simplified path as high likelihood or low likelihood; and
display on the GUI one or more anomalies detected in the log, wherein the one or more anomalies are detected based on the plurality of anomalous log data prediction models and the character heatmap.
16. The system of claim 15 , wherein
the A-replace operation comprises performing regular expression (regex) character replacement for one or more special characters in the plurality of strings by replacing (a) one or more alphanumeric characters, (b) one or more numbers, or (c) one or more upper case characters, one or more low case characters, and one or more numbers, and
performing the regex character replacement identifies one or more common patterns of special characters that comprise globally unique identifiers (GUIDs).
17. The system of claim 15 , wherein the program instructions are executable to:
cause the GUI to identify an anomaly in the log by displaying a sequence of characters in a path in a bolded, underlined, or bigger font, wherein the sequence of characters is determined to be unusual based on the anomalous log data prediction models.
18. The system of claim 15 , wherein the program instructions are executable to:
receive a new process event;
extract a new path from the new process event;
process the new path using the plurality of trained anomalous log data prediction models that are trained using the updated dataset that comprises a plurality of transition probabilities of a plurality of paths;
generate an updated character heatmap that indicates a likelihood of each character in the new path predicted by each of the plurality of trained anomalous log data prediction models; and
generate a regex replacement or normalization for (a) each of one or more new strings in the new path and (b) for one or more existing strings in the path or in other paths in the log that creates a new simplified path for one or more of the plurality of trained anomalous log data prediction models.
19. The system of claim 18 , wherein the program instructions are executable to:
access the updated character heatmap;
compare each of the new simplified paths for each of the plurality of trained anomalous log data prediction models based on the likelihood that each character in the new path is high likelihood or low likelihood; and
based on the comparing, identify one or more new simplified paths of one or more trained anomalous log data prediction models as anomalous.
20. The system of claim 15 , wherein the program instructions are executable to:
receive historical log data;
batch the historical log data into a plurality of paths; and
filter the plurality of paths based on a single type of string of one or more types of strings that are part of the plurality of paths.