Joint semantic and format similarity for large scale log retrieval

1. A method for diagnosing computer system faults using log retrieval based on joint semantic and syntactic similarities, comprising:

receiving a set of query logs;

defining joint semantic and syntactic similarities between the set of query logs and respective ones of multiple sets of historical logs based on semantic content and syntactic information obtained for the set of query logs and the multiple sets of historical logs, the multiple sets of historical logs being associated with historical computer system fault diagnoses;

retrieving a set of historical logs from the multiple sets of historical logs to obtain a retrieved set of historical logs for computer system fault comparison based on a similarity measure corresponding to each of the multiple sets of historical logs derived from the joint semantic and syntactic similarities; and

transmitting the retrieved set of historical logs to one or more computing devices to perform the computer system fault comparison.

2. The method of claim 1 , further comprising:

generating a global log format dictionary by performing adaptive log format learning to generate log syntactic formats;

storing the global log format dictionary in a global log formats database; and

extracting the semantic content for the multiple sets of historical logs based on the global log formats dictionary.

3. The method of claim 2 , wherein extracting the semantic content for the multiple sets of historical logs further includes:

performing log parsing against the global log formats database to obtain parsed logs;

aggregating the parsed logs based on log format identifiers; and

organizing log semantic content for different log formats based on the aggregation.

4. The method of claim 2 , further comprising forming first feature vectors representing respective ones of the multiple sets of historical logs with joint semantic and syntactic representations based on the obtained semantic content and syntactic information, and storing the first feature vectors in a log signature database.

5. The method of claim 4 , further comprising:

parsing each query log of the set of query logs against the global log formats database;

extracting semantic content for each parsed query log; and

forming second feature vectors representing the set of query logs with joint semantic and syntactic representations based on the semantic content and syntactic information for the set of query logs.

6. The method of claim 5 , further comprising computing similarity distances between the first and second feature vectors to generate the similarity measures for the multiple sets of historical logs.

7. The method of claim 1 , wherein retrieving the set of historical logs further includes ranking the multiple sets of historical logs based on the similarity measures, and identifying the set of historical logs for retrieval based on the ranking.

8. The method of claim 7 , wherein retrieving the set of historical logs further includes merging the similarity measures based on a time resolution parameter or a time range.

9. A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for diagnosing computer system faults using log retrieval based on joint semantic and syntactic similarities, the method performed by the computer comprising:

receiving a set of query logs;

defining joint semantic and syntactic similarities between the set of query logs and respective ones of multiple sets of historical logs based on semantic content and syntactic information obtained for the set of query logs and the multiple sets of historical logs, the multiple sets of historical logs being associated with historical computer system fault diagnoses;

retrieving a set of historical logs from the multiple sets of historical logs to obtain a retrieved set of historical logs for computer system fault comparison based on a similarity measure corresponding to each of the multiple sets of historical logs derived from the joint semantic and syntactic similarities; and

transmitting the retrieved set of historical logs to one or more computing devices to perform the computer system fault comparison.

10. The computer program product of claim 9 , wherein the method further includes:

generating a global log format dictionary by performing adaptive log format learning to generate log syntactic formats;

storing the global log format dictionary in a global log formats database; and

extracting the semantic content for the multiple sets of historical logs based on the global log formats dictionary.

11. The computer program product of claim 10 , wherein extracting the semantic content for the multiple sets of historical logs further includes:

performing log parsing against the global log formats database to obtain parsed logs;

aggregating the parsed logs based on log format identifiers; and

organizing log semantic content for different log formats based on the aggregation.

12. The computer program product of claim 10 , wherein the method further includes forming first feature vectors representing respective ones of the multiple sets of historical logs with joint semantic and syntactic representations based on the obtained semantic content and syntactic information, and storing the first feature vectors in a log signature database.

13. The computer program product of claim 12 , wherein the method further includes:

parsing each query log of the set of query logs against the global log formats database;

extracting semantic content for each parsed query log; and

forming second feature vectors representing the set of query logs with joint semantic and syntactic representations based on the semantic content and syntactic information for the set of query logs.

14. The computer program product of claim 13 , further comprising computing similarity distances between the first and second feature vectors to generate the similarity measures for the multiple sets of historical logs.

15. The computer program product of claim 9 , wherein retrieving the set of historical logs further includes ranking the multiple sets of historical logs based on the similarity measures, and identifying the set of historical logs for retrieval based on the ranking.

16. The computer program product of claim 15 , wherein retrieving the set of historical logs further includes merging the similarity measures based on a time resolution parameter or a time range.

17. A system for diagnosing computer system faults using log retrieval based on joint semantic and syntactic similarities, comprising:

a memory device for storing program code; and

at least one processor device operatively coupled to a memory device and configured to execute program code stored on the memory device to:

receive a set of query logs;

define joint semantic and syntactic similarities between the set of query logs and respective ones of multiple sets of historical logs based on semantic content and syntactic information obtained for the set of query logs and the multiple sets of historical logs, the multiple sets of historical logs being associated with historical computer system fault diagnoses;

retrieve a set of historical logs from the multiple sets of historical logs to obtain a retrieved set of historical logs for computer system fault comparison based on a similarity measure corresponding to each of the multiple sets of historical logs derived from the joint semantic and syntactic similarities; and

transmit the retrieved set of historical logs to one or more computing devices to perform the computer system fault comparison.

18. The system of claim 17 , wherein the at least one processor device is further configured to execute program code stored on the memory device to:

generate a global log format dictionary by performing adaptive log format learning to generate log syntactic formats;

store the global log format dictionary in a global log formats database; and

extract the semantic content for the multiple sets of historical logs based on the global log formats dictionary, wherein the at least one processor device is further configured to extract the semantic content for the multiple sets of historical logs by:

performing log parsing against the global log formats database to obtain parsed logs;

aggregating the parsed logs based on log format identifiers; and

organizing log semantic content for different log formats based on the aggregation.

19. The system of claim 18 , wherein the at least one processor device is further configured to execute program code stored on the memory device to:

form first feature vectors representing respective ones of the multiple sets of historical logs with joint semantic and syntactic representations based on the obtained semantic content and syntactic information;

store the first feature vectors in a log signature database;

parse each query log of the set of query logs against the global log formats database;

extract semantic content for each parsed query log;

form second feature vectors representing the set of query logs with joint semantic and syntactic representations based on the semantic content and syntactic information for the set of query logs; and

compute similarity distances between the first and second feature vectors to generate the similarity measures for the multiple sets of historical logs.

20. The system of claim 17 , wherein the at least one processor device is further configured to retrieve the set of historical logs by ranking the multiple sets of historical logs based on the similarity measures, and identifying the set of historical logs for retrieval based on the ranking.