LOG RECORD ANALYSIS USING SIMILARITY DISTRIBUTIONS OF CONTEXTUAL LOG RECORD SERIES
A plurality of textual log records characterizing operations occurring within a technology landscape may be received and converted into numerical log record vectors. For a current log record vector and a preceding set of log record vectors of the numerical log record vectors, a similarity series may be computed that includes a similarity measure for each of a set of log record vector pairs, with each log record vector pair including the current log record vector and one of the preceding set of log record vectors. A similarity distribution of the similarity series may be generated, and an anomaly in the operations occurring within the technology landscape may be detected, based on the similarity distribution.
1 . A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:
receive textual log records characterizing operations occurring within a technology landscape;
convert the textual log records into numerical log record vectors;
compute, for a current log record vector and a preceding set of log record vectors of the numerical log record vectors, a similarity series that includes a similarity measure for each of a set of log record vector pairs, wherein each log record vector pair includes the current log record vector and one of the preceding set of log record vectors;
generate a similarity distribution of the similarity series; and
detect an anomaly in the operations occurring within the technology landscape, based on the similarity distribution.
2 . The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
process the textual log records with an embedding model to convert the textual log records into the numerical log record vectors.
3 . The computer program product of claim 1 , wherein the similarity measure includes a similarity score calculated using a cosine similarity algorithm.
4 . The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
calculate the similarity distribution as a frequency distribution measuring a frequency of each similarity measure of the similarity series.
5 . The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
determine at least one similarity threshold for the similarity distribution; and
determine the anomaly, based on the similarity distribution and the at least one similarity threshold.
6 . The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
determine a similarity trend of the similarity distribution; and
determine the anomaly, based on the similarity distribution and the similarity trend.
7 . The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
determine an earlier similarity distribution for an earlier log record vector and earlier set of log record vectors; and
determine the anomaly, based on a comparison of the similarity distribution and the earlier similarity distribution.
8 . The computer program product of claim 7 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
apply a Kalman filter to the similarity distribution and the earlier similarity distribution to determine the anomaly.
9 . The computer program product of claim 7 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
apply a Kullback-Leibler (KL) divergence analysis to the similarity distribution and the earlier similarity distribution to determine the anomaly.
10 . The computer program product of claim 1 , wherein the instructions, when executed, are further configured to cause the at least one computing device to:
detect the anomaly in a component for which a current log record of the current log record vector was generated.
11 . A computer-implemented method, the method comprising:
receiving textual log records characterizing operations occurring within a technology landscape;
converting the textual log records into numerical log record vectors;
computing, for a current log record vector and a preceding set of log record vectors of the numerical log record vectors, a similarity series that includes a similarity measure for each of a set of log record vector pairs, wherein each log record vector pair includes the current log record vector and one of the preceding set of log record vectors;
generating a similarity distribution of the similarity series; and
detecting an anomaly in the operations occurring within the technology landscape, based on the similarity distribution.
12 . The method of claim 11 , further comprising:
processing the textual log records with an embedding model to convert the textual log records into the numerical log record vectors.
13 . The method of claim 11 , further comprising:
calculating the similarity distribution as a frequency distribution measuring a frequency of each similarity measure of the similarity series.
14 . The method of claim 11 , further comprising:
determining at least one similarity threshold for the similarity distribution; and
determining the anomaly, based on the similarity distribution and the at least one similarity threshold.
15 . The method of claim 11 , further comprising:
determining a similarity trend of the similarity distribution; and
determining the anomaly, based on the similarity distribution and the similarity trend.
16 . The method of claim 11 , further comprising:
determining an earlier similarity distribution for an earlier log record vector and earlier set of log record vectors; and
determining the anomaly, based on a comparison of the similarity distribution and the earlier similarity distribution.
17 . A system comprising:
at least one memory including instructions; and
at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the at least one processor to:
receive textual log records characterizing operations occurring within a technology landscape;
convert the textual log records into numerical log record vectors;
compute, for a current log record vector and a preceding set of log record vectors of the numerical log record vectors, a similarity series that includes a similarity measure for each of a set of log record vector pairs, wherein each log record vector pair includes the current log record vector and one of the preceding set of log record vectors;
generate a similarity distribution of the similarity series; and
detect an anomaly in the operations occurring within the technology landscape, based on the similarity distribution.
18 . The system of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
process the textual log records with an embedding model to convert the textual log records into the numerical log record vectors.
19 . The system of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
determine at least one similarity threshold for the similarity distribution; and
determine the anomaly, based on the similarity distribution and the at least one similarity threshold.
20 . The system of claim 17 , wherein the instructions, when executed, are further configured to cause the at least one processor to:
determine an earlier similarity distribution for an earlier log record vector and earlier set of log record vectors; and
determine the anomaly, based on a comparison of the similarity distribution and the earlier similarity distribution.