Log sampling and storage system
A log sampling and storage system reduces volumes of stored log information. A log storage engine receives a series of log messages responsive to a request. The engine compares that series to various patterns previously generated from historical log data. This comparison can reference •a name of the request; •a total time to process the request; •an item count; •a message length; and/or •a message hash code. Comparing incoming log series with the predefined patterns, results in generation of a similarity score. If the score falls outside a tolerance, the existence of an outlier log message series is indicated, and that outlier log message series is stored in its entirety. However, if the similarity score falls within a tolerance, similarity to an existing predefined pattern is indicated. Rather than storing the (non-outlier) log message series, the engine directs storing a link to a sample log message series reflecting the pattern.
1. A method comprising:
receiving a log series comprising a plurality of messages each having an associated processing time cost;
extracting the log series into a first pattern, the first pattern comprising:
a request name;
a first log message;
a second log message; and
total processing time cost information corresponding to a total processing time cost of the plurality of messages;
comparing the first pattern to a second pattern stored in a non-transitory computer readable storage medium to generate a similarity score;
storing statistics in the non-transitory computer readable storage medium,
wherein the statistics comprise an ongoing average of total processing time costs of log series previously matched with the second pattern;
determining that the similarity score lies within a threshold value, the similarity score comprising at least a comparison of the total processing time cost information with the ongoing average of total processing time costs; and
when the similarity score is outside the threshold value, storing the received log series, and
when the similarity score is within the threshold value, storing a link to an existing sampled log series previously matching the second pattern, not storing the received log series, and updating the ongoing average of the total processing time costs.
2. A method as in claim 1 further comprising:
sampling a history of log series data to identify a plurality of patterns including the second pattern; and
storing the plurality of patterns in the non-transitory computer readable storage medium.
3. A method as in claim 1 wherein the cost information comprises a total time of execution.
4. A method as in claim 3 wherein the cost information further comprises:
a time of execution of the first log message; and
a time of execution of the second log message.
5. A method as in claim 1 wherein statistics further comprise a count for the second pattern.
6. A method as in claim 1 wherein the statistics further comprise updated cost information of the second pattern.
7. A method as in claim 1 wherein the comparing indicates that an exception is not present in the first log series.
8. A method as in claim 1 wherein:
each message of the first pattern comprises a separate item; and
the comparing indicates that an item count of the first pattern matches an item count of the second pattern.
9. A method as in claim 1 wherein:
the non-transitory computer readable storage medium comprises an in-memory database; and
the comparing is performed by an in-memory database engine of the in-memory database.
10. A non-transitory computer readable storage medium embodying a computer program for performing a method, said method comprising:
receiving a log series comprising a plurality of messages each having an associated processing time cost;
extracting the log series into a first pattern, the first pattern comprising:
a request name;
a first log message;
a second log message; and
total processing time cost information corresponding to a total processing time cost of the plurality of messages;
comparing the first pattern to a second pattern stored in a non-transitory computer readable storage medium to generate a similarity score;
storing statistics in the non-transitory computer readable storage medium,
wherein the statistics comprise an ongoing average of total processing time costs of log series previously matched with the second pattern;
determining that the similarity score lies within a threshold value, the similarity score comprising at least a comparison of the total processing time cost information with the ongoing average of total processing time costs; and
when the similarity score is outside the threshold value, storing the received log series, and
when the similarity score is within the threshold value, storing a link to an existing sampled log series previously matching the second pattern, not storing the received log series, and updating the ongoing average of the total processing time costs.
11. A non-transitory computer readable storage medium as in claim 10 wherein the cost information further comprises:
a time of execution of the first log message; and
a time of execution of the second log message.
12. A non-transitory computer readable storage medium as in claim 10 wherein the method further comprises:
updating the statistics including a count for the second pattern.
13. A non-transitory computer readable storage medium as in claim 10 wherein the statistics further comprise updated cost information of the second pattern.
14. A non-transitory computer readable storage medium as in claim 10 wherein the comparing indicates that an exception is not present in the first log series.
15. A computer system comprising:
one or more processors;
a software program, executable on said computer system, the software program configured to cause an in-memory database engine of an in-memory source database to:
receive a log series comprising a plurality of messages each having an associated processing time cost;
extract the log series into a first pattern, the first pattern comprising:
a request name;
a first log message;
a second log message; and
total processing time cost information corresponding to a total processing time cost of the plurality of messages;
compare the first pattern to a second pattern stored in the in-memory database to generate a similarity score;
store statistics in the in-memory database, wherein the statistics comprise an ongoing average of total processing time costs of log series previously matched with the second pattern;
determine that the similarity score lies within a threshold value the similarity score comprising at least a comparison of the total processing time cost information with the ongoing average of total processing time costs; and
when the similarity score is outside the threshold value, store the received log series, and
when the similarity score is within the threshold value, store a link to an existing sampled log series previously matching the second pattern not storing the received log series, and update the ongoing average of the total processing time costs.
16. A computer system as in claim 15 wherein the cost information comprises:
a time of execution of the first log message;
a time of execution of the second log message; and
a total time of execution.
17. A computer system as in claim 15 wherein the in-memory database engine is further configured to:
update the statistics including a count for the second pattern and cost information of the second pattern.
18. A computer system as in claim 15 wherein the comparing indicates that an exception is not present in the first log series.
19. A computer system as in claim 15 wherein:
each message of the first pattern comprises a separate item; and
the comparing indicates that an item count of the first pattern matches an item count of the second pattern.
20. A computer system as in claim 15 wherein the in-memory database engine is further configured to:
sample a history of log series data to identify a plurality of patterns including the second pattern; and
store the plurality of patterns in the in-memory database.