Character-level log parsing

1 . A computing system comprising:

memory;

one or more hardware processors coupled to the memory; and

one or more non-transitory computer readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations comprising:

receiving a log message comprising a sequence of L characters, wherein L is a positive integer;

generating embeddings for the sequence of L characters;

generating feature vectors for N tokens based on the embeddings, wherein a token represents M consecutive characters in the log message, wherein M is an integer greater than one, and N is a smallest integer greater than or equal to L divided by M;

predicting N binary coded values based on the feature vectors for N tokens;

generating a sequence of L parameter masks based on the N binary coded values, wherein a parameter mask indicates that a corresponding character in the log message is a static character or a variable character; and

generating, using the sequence of L parameter masks, a log template as a structured representation of the log message.

2 . The computing system of claim 1 , wherein the log template retains static characters from the log message and replaces variable characters in the log message with one or more placeholders, wherein static characters and placeholders in the log template are positionally aligned with their corresponding characters in the log message.

3 . The computing system of claim 1 , wherein generating the sequence of L parameter masks comprises mapping each one of the N binary coded values into parameter masks for M consecutive characters within a corresponding token.

4 . The computing system of claim 1 , wherein the operations further comprises padding the log message with one or more predefined padding characters responsive to determining that L is not a multiple of M.

5 . The computing system of claim 1 , wherein generating feature vectors for N tokens comprises converting the embeddings into encoded feature vectors for the sequence of L characters, wherein the converting is performed by an encoder comprising a self-attention neural network configured to measure contextual dependencies between characters in the log message.

6 . The computing system of claim 5 , wherein generating feature vectors for N tokens further comprises down-sampling the encoded feature vectors for the sequence of L characters using a one-dimensional convolutional neural network.

7 . The computing system of claim 6 , wherein the one-dimensional convolutional neural network has a non-overlapping kernel of size M and a stride of M.

8 . The computing system of claim 1 , wherein predicting N binary coded values comprises processing the feature vectors for N tokens using a bidirectional long short-term memory neural network to generate sequential feature representations for the N tokens.

9 . The computing system of claim 8 , wherein predicting N binary coded values further comprises processing the sequential feature representations for the N tokens using a conditional random field neural network.

10 . The computing system of claim 9 , wherein the operations further comprise training the conditional random field neural network using a labelled dataset comprising a plurality of input log messages and a plurality of ground truth log templates corresponding to the plurality of input log messages, wherein the training uses a negative log-likelihood loss function to minimize a prediction error in classifying static and variable characters within the plurality of input log messages based on the plurality of ground truth log templates.

11 . A computer-implemented method comprising:

receiving a log message comprising a sequence of L characters, wherein L is a positive integer;

generating embeddings for the sequence of L characters;

generating feature vectors for N tokens based on the embeddings, wherein a token represents M consecutive characters in the log message, wherein M is an integer greater than one, and N is a smallest integer greater than or equal to L divided by M;

predicting N binary coded values based on the feature vectors for N tokens;

generating a sequence of L parameter masks based on the N binary coded values, wherein a parameter mask indicates that a corresponding character in the log message is a static character or a variable character; and

generating, using the sequence of L parameter masks, a log template as a structured representation of the log message.

12 . The computer-implemented method of claim 11 , wherein the log template retains static characters from the log message and replaces variable characters in the log message with one or more placeholders, wherein static characters and placeholders in the log template are positionally aligned with their corresponding characters in the log message.

13 . The computer-implemented method of claim 11 , wherein generating the sequence of L parameter masks comprises mapping each one of the N binary coded values into parameter masks for M consecutive characters within a corresponding token.

14 . The computer-implemented method of claim 11 , further comprising padding the log message with one or more predefined padding characters responsive to determining that L is not a multiple of M.

15 . The computer-implemented method of claim 11 , wherein generating feature vectors for N tokens comprises converting the embeddings into encoded feature vectors for the sequence of L characters, wherein the converting is performed by an encoder comprising a self-attention neural network configured to measure contextual dependencies between characters in the log message.

16 . The computer-implemented method of claim 15 , wherein generating feature vectors for N tokens further comprises down-sampling the encoded feature vectors for the sequence of L characters using a one-dimensional convolutional neural network.

17 . The computer-implemented method of claim 16 , wherein the one-dimensional convolutional neural network has a non-overlapping kernel of size M and a stride of M.

18 . The computer-implemented method of claim 11 , wherein predicting N binary coded values comprises processing the feature vectors for N tokens using a bidirectional long short-term memory neural network to generate sequential feature representations for the N tokens.

19 . The computer-implemented method of claim 18 , wherein predicting N binary coded values further comprises processing the sequential feature representations for the N tokens using a conditional random field neural network.

20 . One or more non-transitory computer-readable media having encoded thereon computer-executable instructions causing one or more processors to perform a method, the method comprising:

receiving a log message comprising a sequence of L characters, wherein L is a positive integer;

generating embeddings for the sequence of L characters;

generating feature vectors for N tokens based on the embeddings, wherein a token represents M consecutive characters in the log message, wherein M is an integer greater than one, and N is a smallest integer greater than or equal to L divided by M;

predicting N binary coded values based on the feature vectors for N tokens;

generating a sequence of L parameter masks based on the N binary coded values, wherein a parameter mask indicates that a corresponding character in the log message is a static character or a variable character; and

generating, using the sequence of L parameter masks, a log template as a structured representation of the log message.