Correlation of time series signals using shape identification values
Some embodiments relate to analyzing log records. A method may include determining a first shape identification value that characterizes a shape described by data points of a first time series signal that represents a time distribution of timestamps of a first plurality of messages. For each message among a second plurality of messages, the method may also include determining a shape identification value for the message that characterizes a shape described by data points of a corresponding time series signal that represents a time distribution of timestamps of a plurality of instances of the message. The method may further include determining that a shape identification value, from among the shape identification values for the second plurality of messages, is the same as the first shape identification value and, in response to the determining, providing information identifying the corresponding message to a user interface.
1 . A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
identifying a message portion and a time period;
obtaining, from at least one of a plurality of log files, a first plurality of messages that include the message portion and have a timestamp within the time period;
generating a first time series signal that represents a time distribution of the timestamps of the first plurality of messages;
determining a first shape identification value that characterizes a shape described by data points of the first time series signal;
for each message among a second plurality of messages:
obtaining, from at least one of the plurality of log files, a corresponding plurality of instances of the message that have a timestamp within the time period;
generating a corresponding time series signal that represents a time distribution of the timestamps of the plurality of instances of the message; and
determining a shape identification value for the message that characterizes a shape described by data points of the corresponding time series signal; and
for each of at least one message among the second plurality of messages:
determining that a shape identification value for the message is the same as the first shape identification value; and
in response to the determining, providing information identifying the message to a user interface.
2 . The computer-program product as recited in claim 1 , wherein the message portion comprises a text string.
3 . The computer-program product as recited in claim 1 , wherein each of the data points of the first time series signal corresponds to a respective time bin of a plurality of time bins and indicates a quantity of the timestamps of the first plurality of messages that are within the respective time bin.
4 . The computer-program product as recited in claim 1 , wherein:
for one of the messages among the second plurality of messages, obtaining the corresponding plurality of instances of the message comprises obtaining the corresponding plurality of instances of the message from a first log file, and
for another one of the messages among the second plurality of messages, obtaining the corresponding plurality of instances of the message comprises obtaining the corresponding plurality of instances of the message from a second log file that is separate from the first log file.
5 . The computer-program product as recited in claim 1 , wherein:
determining the shape identification value for one of the messages among the second plurality of messages is performed at a first host, and
determining the shape identification value for another one of the messages among the second plurality of messages is performed at a second host that is separate from the first host.
6 . The computer-program product as recited in claim 1 , wherein the first shape identification value is based on a persistence score signal of the first time series signal.
7 . The computer-program product as recited in claim 6 , wherein determining the first shape identification value comprises reducing a magnitude resolution of the persistence score signal, based on a number of peaks in the persistence score signal.
8 . The computer-program product as recited in claim 6 , wherein determining the first shape identification value comprises reducing a resolution of the persistence score signal in both magnitude and time, based on a number of peaks in the persistence score signal.
9 . The computer-program product as recited in claim 6 , wherein determining the first shape identification value comprises applying a hash function to a key that is based on the persistence score signal.
10 . The computer-program product as recited in claim 1 , wherein the first shape identification value is based on a peak persistence score signal of the first time series signal and a valley persistence score signal of the first time series signal.
11 . A computer-implemented method of analyzing log records, the method comprising:
identifying a text string and a time period;
obtaining, from at least one of a plurality of log files, a first plurality of messages that include the text string and have a timestamp within the time period;
generating a first time series signal that represents a time distribution of the timestamps of the first plurality of messages;
determining a first shape identification value that characterizes a shape described by data points of the first time series signal;
for each message among a second plurality of messages:
obtaining, from at least one of the plurality of log files, a corresponding plurality of instances of the message that have a timestamp within the time period;
generating a corresponding time series signal that represents a time distribution of the timestamps of the plurality of instances of the message; and
determining a shape identification value for the message that characterizes a shape described by data points of the corresponding time series signal; and
for each of at least one message among the second plurality of messages:
determining that a shape identification value for the message is the same as the first shape identification value; and
in response to the determining, providing information identifying the message to a user interface.
12 . The computer-implemented method as recited in claim 11 , wherein:
determining the shape identification value for one of the messages among the second plurality of messages is performed at a first host, and
determining the shape identification value for another one of the messages among the second plurality of messages is performed at a second host that is separate from the first host.
13 . The computer-implemented method as recited in claim 11 , wherein the first shape identification value is based on a persistence score signal of the first time series signal.
14 . The computer-implemented method as recited in claim 13 , wherein determining the first shape identification value comprises reducing a resolution of the persistence score signal in both magnitude and time, based on a number of peaks in the persistence score signal.
15 . The computer-implemented method as recited in claim 11 , wherein the first shape identification value is based on a peak persistence score signal of the first time series signal and a valley persistence score signal of the first time series signal.
16 . A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
identifying a text string and a time period;
obtaining, from at least one of a plurality of log files, a first plurality of messages that include the text string and have a timestamp within the time period;
generating a first time series signal that represents a time distribution of the timestamps of the first plurality of messages;
determining a first shape identification value that characterizes a shape described by data points of the first time series signal;
for each message among a second plurality of messages:
obtaining, from at least one of the plurality of log files, a corresponding plurality of instances of the message that have a timestamp within the time period;
generating a corresponding time series signal that represents a time distribution of the timestamps of the plurality of instances of the message; and
determining a shape identification value for the message that characterizes a shape described by data points of the corresponding time series signal; and
for each of at least one message among the second plurality of messages:
determining that a shape identification value for the message is the same as the first shape identification value; and
in response to the determining, providing information identifying the message to a user interface.
17 . The system as recited in claim 16 , wherein:
for one of the messages among the second plurality of messages, obtaining the corresponding plurality of instances of the message comprises obtaining the corresponding plurality of instances of the message from a first log file, and
for another one of the messages among the second plurality of messages, obtaining the corresponding plurality of instances of the message comprises obtaining the corresponding plurality of instances of the message from a second log file that is separate from the first log file.
18 . The system as recited in claim 16 , wherein:
determining the shape identification value for one of the messages among the second plurality of messages is performed at a first host, and
determining the shape identification value for another one of the messages among the second plurality of messages is performed at a second host that is separate from the first host.
19 . The system as recited in claim 16 , wherein:
the first shape identification value is based on a persistence score signal of the first time series signal, and
determining the first shape identification value comprises reducing a resolution of the persistence score signal, based on a number of peaks in the persistence score signal, in at least one of magnitude or time.
20 . The system as recited in claim 16 , wherein the first shape identification value is based on a peak persistence score signal of the first time series signal and a valley persistence score signal of the first time series signal.