Computer system diagnostic log chain
Disclosed embodiments provide a computer-implemented technique for creating a log chain from multiple software component logs. The log chain can include log information from multiple software components that interact with each other. In embodiments, a locality-sensitive hashing technique is used to create a signature of a log chain. Machine-learning systems are trained with log chains generated from test execution, and/or log chains from incidents with deployed software. In embodiments, logs are pre-processed by tokenizing and trimming. Logs from various components that interact with each other may be combined into a temporally sequential log chain. The signature of the log chain may be used to identify additional information about the error from a machine-learning process that was trained on previously generated log chains. In this way, the time required to identify a problem can be significantly reduced, resulting in increased reliability and availability of complex computer systems.
1. A computer-implemented method for log processing, comprising:
obtaining test case logs having log lines resulting from a computer operation for a test case from a set of devices executing software;
identifying a marker in the test case logs, the marker being a pattern that is indicative of a problem;
performing a log trimming operation on the test case logs that discards log lines that are not based on the identified marker; and
creating a log chain of log lines specific to the problem, wherein the log chain comprises a remainder of the log lines from the plurality of the test case logs that remain from the log trimming operation in chronological order.
2. The computer-implemented method of claim 1 , wherein the log chain comprises logs from at least three components.
3. The computer-implemented method of claim 2 , wherein the log chain comprises logs from less than 200 components.
4. The computer-implemented method of claim 1 , wherein the log chain comprises logs from two adjacent components.
5. The computer-implemented method of claim 1 , further comprising filtering the log chain based on a thread identifier.
6. The computer-implemented method of claim 1 , further comprising computing an information signature of the log chain using a locality-sensitive hashing technique.
7. The computer-implemented method of claim 6 , wherein the locality-sensitive hashing technique comprises a SimHash process.
8. The computer-implemented method of claim 6 , wherein the locality-sensitive hashing technique comprises a minhash process.
9. The computer-implemented method of claim 1 , further comprising:
computing a first set of information signatures of a first set of log chains using a locality-sensitive hashing technique;
performing a supervised machine learning process using a plurality of information signatures corresponding to a first set of log chains;
computing a second set of information signatures of a second set of log chains using the locality-sensitive hashing technique;
performing a computerized classification of the second set of log chains via a machine learning process; and
generating a correlation report, wherein the correlation report includes one or more log chains from the first set of log chains, and one or more log chains from the second set of log chains, and an indication of a correlation between the one or more log chains from the first set of log chains, and the one or more log chains from the second set of log chains.
10. The computer-implemented method of claim 9 , wherein the locality-sensitive hashing technique comprises a SimHash process.
11. The computer-implemented method of claim 9 , wherein the locality-sensitive hashing technique comprises a minhash process.
12. An electronic computation device comprising:
a processor;
a memory coupled to the processor, the memory containing instructions, that when executed by the processor, cause the electronic computation device to:
obtain test case logs having log lines resulting from a computer operation for a test case from a set of devices executing software;
identify a marker in the test case logs, the marker being a pattern that is indicative of a problem;
perform a log trimming operation on the test case logs that discards log lines that are not based on the identified marker; and
create a log chain of log lines specific to the problem, wherein the log chain comprises a remainder of the log lines from the plurality of the test case logs that remain from the log trimming operation in chronological order.
13. The electronic computation device of claim 12 wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to compute an information signature of the log chain using a locality-sensitive hashing technique.
14. The electronic computation device of claim 13 wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to compute the information signature using a SimHash process.
15. The electronic computation device of claim 13 wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to compute the information signature using a minhash process.
16. The electronic computation device of claim 12 wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to:
compute a first set of information signatures of a first set of log chains using a locality-sensitive hashing technique;
perform a supervised machine learning process using a plurality of information signatures corresponding to a first set of log chains;
compute a second set of information signatures of a second set of log chains using the locality-sensitive hashing technique;
perform a computerized classification of the second set of log chains via a machine learning process; and
generate a correlation report, wherein the correlation report includes one or more log chains from the first set of log chains, and one or more log chains from the second set of log chains, and an indication of a correlation between the one or more log chains from the first set of log chains, and the one or more log chains from the second set of log chains.
17. A computer program product for an electronic computation device comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the electronic computation device to:
obtain test case logs having log lines resulting from a computer operation for a test case from a set of devices executing software;
identify a marker in the test case logs, the marker being a pattern that is indicative of a problem;
perform a log trimming operation on the test case logs that discards log lines that are not based on the identified marker; and
create a log chain of log lines specific to the problem, wherein the log chain comprises a remainder of the log lines from the plurality of the test case logs that remain from the log trimming operation in chronological order.
18. The computer program product of claim 17 , wherein the computer readable storage medium includes program instructions executable by the processor to cause the electronic computation device to compute an information signature of the log chain using a locality-sensitive hashing technique.
19. The computer program product of claim 18 , wherein the computer readable storage medium includes program instructions executable by the processor to cause the electronic computation device to compute the information signature using a SimHash process.
20. The computer program product of claim 17 , wherein the computer readable storage medium includes program instructions executable by the processor to cause the electronic computation device to:
compute a first set of information signatures of a first set of log chains using a locality-sensitive hashing technique;
perform a supervised machine learning process using a plurality of information signatures corresponding to a first set of log chains;
compute a second set of information signatures of a second set of log chains using the locality-sensitive hashing technique;
perform a computerized classification of the second set of log chains via a machine learning process; and
generate a correlation report, wherein the correlation report includes one or more log chains from the first set of log chains, and one or more log chains from the second set of log chains, and an indication of a correlation between the one or more log chains from the first set of log chains, and the one or more log chains from the second set of log chains.