ERROR DEDUPLICATION AND REPORTING FOR A DATA MANAGEMENT SYSTEM BASED ON NATURAL LANGUAGE PROCESSING OF ERROR MESSAGES
Methods, systems, and devices for data management are described. A data management system (DMS) may generate or receive error logs when a backup process fails. To more easily analyze and compare error logs, the error logs may be processed by a natural language processing (NLP) program. The NLP program extracts, from an individual error log, a natural language string for the error message and metadata associated with the error. The processed error logs may be stored in a database as a natural language string for the error message and metadata extracted from the error log by the NLP program. The NLP program may simplify the string by removing items such as punctuation, defined stop words, geographic terms, uniform resource locators, gerunds, and tenses, may tokenize the string, and extract metadata type information. Comparison and analysis may more easily be performed on the natural language error strings and accompanying metadata.
1 . A method, comprising:
receiving, from a first database, a set of error logs associated with a data management system;
performing natural language processing on the set of error logs to generate a set of natural language error strings and corresponding metadata for the set of error logs;
storing the set of natural language error strings and corresponding metadata for the set of error logs in a second database; and
generating an error report based on the set of natural language error strings and corresponding metadata stored in the second database, wherein the error report identifies one or more same errors associated with the set of error logs based on the set of natural language error strings.
2 . The method of claim 1 , wherein generating the error report comprises:
generating the error report identifying a set of unique errors associated with the set of error logs, wherein a same error in two or more error logs of the set of error logs comprises a single unique error.
3 . The method of claim 2 , wherein generating the error report comprises:
generating the error report indicating a quantity of instances each unique error occurred in the set of error logs.
4 . The method of claim 1 , further comprising:
identifying that two or more natural language error strings correspond to a same error based on the two or more natural language error strings having a character similarity satisfying a threshold.
5 . The method of claim 1 , wherein generating the error report comprises:
generating the error report based on a duration since generation of a previous error report satisfying a threshold.
6 . The method of claim 5 , further comprising:
receiving, from a user interface view associated with an administrator of the data management system, an indication of the threshold.
7 . The method of claim 5 , wherein generating the error report comprises:
generating the error report based on additional natural language error strings and additional corresponding metadata associated with additional sets of error logs stored in the second database since the previous error report, wherein sets of error logs are periodically received, and wherein natural language processing is periodically performed on the sets of error logs that are periodically received to generate sets of natural language error strings and corresponding metadata for the set of error logs that are periodically received, and wherein the generated sets of natural language error strings and corresponding metadata for the set of error logs that are periodically received are periodically stored in the second database.
8 . The method of claim 1 , wherein generating the error report comprises:
generating the error report based on a quantity of instances of a same error satisfying a threshold.
9 . The method of claim 8 , further comprising:
receiving, from a user interface view associated with an administrator of the data management system, an indication of the threshold.
10 . The method of claim 1 , wherein generating the error report comprises:
generating the error report based on the corresponding metadata satisfying a triggering condition, the triggering condition comprising an account identifier, an object identifier, a job type, or a customer identifier.
11 . The method of claim 1 , wherein performing the natural language processing on the set of error logs comprises:
removing punctuation in error strings of the set of error logs, removing stop words in the error strings of the set of error logs, removing numerals in error strings of the set of error logs, removing uniform resource locators in the error strings of the set of error logs, removing geographic information in the error strings of the set of error logs, performing tokenization on the error strings of the set of error logs, performing stemming on the error strings of the set of error logs, performing lemmatization on the error strings of the set of error logs, or a combination thereof.
12 . The method of claim 11 , further comprising:
receiving, from a user interface view associated with an administrator of the data management system, an indication of the stop words.
13 . The method of claim 1 , further comprising:
presenting, at user interface view associated with an administrator of the data management system, the error report.
14 . The method of claim 1 , wherein the first database and the second database comprise a same database.
15 . The method of claim 1 , wherein the first database is different from the second database.
16 . An apparatus, comprising:
a processor;
memory coupled with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
receive, from a first database, a set of error logs associated with a data management system;
perform natural language processing on the set of error logs to generate a set of natural language error strings and corresponding metadata for the set of error logs;
store the set of natural language error strings and corresponding metadata for the set of error logs in a second database; and
generate an error report based on the set of natural language error strings and corresponding metadata stored in the second database, wherein the error report identifies one or more same errors associated with the set of error logs based on the set of natural language error strings.
17 . The apparatus of claim 16 , wherein the instructions to generate the error report are executable by the processor to cause the apparatus to:
generate the error report identifying a set of unique errors associated with the set of error logs, wherein a same error in two or more error logs of the set of error logs comprises a single unique error.
18 . The apparatus of claim 17 , wherein the instructions to generate the error report are executable by the processor to cause the apparatus to:
generate the error report indicating a quantity of instances each unique error occurred in the set of error logs.
19 . The apparatus of claim 16 , wherein the instructions are further executable by the processor to cause the apparatus to:
identify that two or more natural language error strings correspond to a same error based on the two or more natural language error strings having a character similarity satisfying a threshold.
20 . A non-transitory computer-readable medium storing code, the code comprising instructions executable by a processor to:
receive, from a first database, a set of error logs associated with a data management system;
perform natural language processing on the set of error logs to generate a set of natural language error strings and corresponding metadata for the set of error logs;
store the set of natural language error strings and corresponding metadata for the set of error logs in a second database; and
generate an error report based on the set of natural language error strings and corresponding metadata stored in the second database, wherein the error report identifies one or more same errors associated with the set of error logs based on the set of natural language error strings.