Systems and methods for parsing log files using classification and a plurality of neural networks
The present disclosure relates to systems and methods for parsing unstructured data with neural networks. In one implementation, a system for parsing unstructured data may include at least one processor and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the system to: receive unstructured data; apply a classifier to the unstructured data to identify a type of the unstructured data; based on the identification, select a corresponding neural network; apply the selected neural network to the unstructured data to obtain structured data; and output the structured data.
1. A system for parsing unstructured data, comprising:
at least one processor; and
at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving unstructured data;
searching the unstructured data for one or more keys associated with one or more types of the unstructured data;
identifying one or more candidate neural networks based on a frequency of the one or more keys within the unstructured data;
extracting one or more feature vectors from the unstructured data;
comparing the one or more extracted feature vectors with one or more representative feature vectors, wherein the one or more representative feature vectors are associated with the one or more candidate neural networks;
based on the comparison, selecting a corresponding neural network from the one or more candidate neural networks;
applying the selected neural network to the unstructured data to obtain structured data; and
outputting the structured data.
2. They system of claim 1 , wherein the operations further comprise training the selected corresponding neural network using a character window.
3. The system of claim 2 , wherein the character window is of a set size.
4. The system of claim 2 , wherein the character window is of a size within a predetermined range of sizes.
5. The system of claim 1 , wherein the one or more keys comprise at least one of a predetermined character or a predetermined pattern.
6. The system of claim 5 , wherein the one or more keys comprise at least one alphanumeric string.
7. The system of claim 1 , wherein the operations further comprise training the selected corresponding neural network using character-by-character analysis.
8. A system for parsing unstructured data, comprising:
at least one processor; and
at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving a log file comprising unstructured data;
searching the unstructured data for one or more keys associated with one or more types of the unstructured data;
identifying one or more candidate neural networks based on a frequency of the one or more keys within the unstructured data;
extracting a distribution of one or more characters;
comparing the extracted distribution with one or more representative distributions, wherein the one or more representative distributions are associated with the one or more candidate neural networks;
based on the comparison, selecting a corresponding neural network from the one or more candidate neural networks;
applying the selected neural network to the log file to obtain structured data corresponding to the unstructured data of the log file; and
outputting the structured data.
9. The system of claim 8 , wherein the operations further comprise training the selected corresponding neural network using a character window.
10. The system of claim 9 , wherein the character window is of a set size.
11. The system of claim 9 , wherein the character window is of a size within a predetermined range of sizes.
12. The system of claim 8 , wherein the one or more keys comprise at least one of a predetermined character or a predetermined pattern.
13. The system of claim 8 , wherein the one or more keys comprise at least one alphanumeric string.
14. The system of claim 8 , wherein the operations further comprise training the selected corresponding neural network using character-by-character analysis.
15. A system for parsing unstructured data, comprising:
at least one processor; and
at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving a log file comprising unstructured data;
searching the unstructured data for one or more keys associated with one or more types of the unstructured data;
identifying one or more candidate neural networks based on a frequency of the one or more keys within the unstructured data;
extracting an application name having generated the log file;
comparing the extracted application name with one or more representative application names, wherein the one or more representative application names are associated with the one or more candidate neural networks;
based on the comparison, selecting a corresponding neural network from the one or more candidate neural networks;
applying the selected neural network to the log file to obtain structured data corresponding to the unstructured data of the log file; and
outputting the structured data.
16. The system of claim 15 , wherein the operations further comprise training the selected corresponding neural network using a character window.
17. The system of claim 16 , wherein the character window is of a set size.
18. The system of claim 16 , wherein the character window is of a size within a predetermined range of sizes.
19. The system of claim 15 , wherein the one or more keys comprise at least one of a predetermined character or a predetermined pattern.
20. The system of claim 15 , wherein the operations further comprise training the selected corresponding neural network using character-by-character analysis.