Automatic parser generation
Automatically generating a parser is disclosed. Raw data is received from a first remote device. A determination that the raw data does not, within a predefined confidence measure, conform to any rules included in a set of rules is made. A clustering function is performed on the raw data. At least one parser rule is generated based on the clustering.
1. A system, comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein instructions provided by the memory to the one or more processors, when executed, cause the one or more processors to:
receive raw data from a first remote device;
determine a plurality of confidence measures for the received raw data with respect to a plurality of existing types of parsers, wherein each confidence measure in the determined plurality of confidence measures comprises a confidence measure for the received raw data with respect to a given type of parser, and wherein a parser is associated with a set of rules included in a library; and
in response to determining that none of the determined plurality of confidence measures with respect to the plurality of existing types of parsers exceeds a threshold, automatically generate a parser applicable to the raw data at least in part by performing a clustering function on the received raw data, wherein automatically generating the parser comprises automatically generating at least one parser rule, wherein automatically generating the at least one parser rule includes generating a regular expression, and wherein the automatically generated at least one parser rule is stored in the library.
2. The system of claim 1 wherein the instructions, when executed, cause the one or more processors to present the generated at least one parser rule to an administrator of the first remote device.
3. The system of claim 2 wherein the instructions, when executed, cause the one or more processors to receive a confirmation from the administrator that the generated at least one parser rule is correct.
4. The system of claim 2 wherein the instructions, when executed, cause the one or more processors to receive a modified version of the generated at least one parser rule from the administrator.
5. The system of claim 1 wherein the instructions, when executed, cause the one or more processors to:
ask an administrator of the first remote device to provide a name of a source of the raw data; and
associate the name with the generated at least one parser rule.
6. The system of claim 1 wherein the instructions, when executed, cause the one or more processors to receive raw data from a second remote device.
7. The system of claim 6 wherein the instructions, when executed, cause the one or more processors to evaluate the raw data received from the second remote device against the generated at least one parser rule.
8. The system of claim 6 wherein the raw data received from the second remote device has an undefined source type.
9. A method, comprising:
receiving, via one or more interfaces, raw data from a first remote device;
determining a plurality of confidence measures for the received raw data with respect to a plurality of existing types of parsers, wherein each confidence measure in the determined plurality of confidence measures comprises a confidence measure for the received raw data with respect to a given type of parser, and wherein a parser is associated with a set of rules included in a library; and
in response to determining that none of the determined plurality of confidence measures with respect to the plurality of existing types of parsers exceeds a threshold, automatically generating a parser applicable to the raw data at least in part by performing, using one or more processors, a clustering function on the received raw data, wherein automatically generating the parser comprises automatically generating at least one parser rule, wherein automatically generating the at least one parser rule includes generating a regular expression, and wherein the automatically generated at least one parser rule is stored in the library.
10. The method of claim 9 further comprising presenting, using the one or more processors, the generated at least one parser rule to an administrator of the first remote device.
11. The method of claim 10 further comprising receiving, via the one or more interfaces, a confirmation from the administrator that the generated at least one parser rule is correct.
12. The method of claim 10 further comprising receiving, via the one or more interfaces, a modified version of the generated at least one parser rule from the administrator.
13. The method of claim 9 further comprising:
asking, using the one or more processors, an administrator of the first remote device to provide a name of a source of the raw data; and
associating, using the one or more processors, the name with the generated at least one parser rule.
14. The method of claim 9 further comprising receiving, via the one or more interfaces, raw data from a second remote device.
15. The method of claim 14 further comprising evaluating, using the one or more processors, the raw data received from the second remote device against the generated at least one parser rule.
16. The method of claim 14 wherein the raw data received from the second remote device has an undefined source type.
17. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, via one or more interfaces, raw data from a first remote device;
determining a plurality of confidence measures for the received raw data with respect to a plurality of existing types of parsers, wherein each confidence measure in the determined plurality of confidence measures comprises a confidence measure for the received raw data with respect to a given type of parser, and wherein a parser is associated with a set of rules included in a library; and
in response to determining that none of the determined plurality of confidence measures with respect to the plurality of existing types of parsers exceeds a threshold, automatically generating a parser applicable to the raw data at least in part by performing, using one or more processors, a clustering function on the received raw data, wherein automatically generating the parser comprises automatically generating at least one parser rule, wherein automatically generating the at least one parser rule includes generating a regular expression, and wherein the automatically generated at least one parser rule is stored in the library.