Automatic parser generation
Automatically generating a parser is disclosed. Raw data is received from a first remote device. A determination that the raw data does not, within a predefined confidence measure, conform to any rules included in a set of rules is made. A clustering function is performed on the raw data. At least one parser rule is generated based on the clustering.
1. A system, comprising:
a set of one or more processors; and
a memory coupled to the set of one or more processors, wherein instructions provided by the memory to the set of one or more processors, when executed, cause the set of one or more processors to:
receive raw data from a first remote device;
determine that the raw data does not, within a predefined confidence measure, conform to any rules included in a set of rules;
perform a clustering function on the raw data;
generate at least one parser rule based at least in part on the clustering;
store, in a data store, the generated at least one parser rule;
receive additional raw data from a second remote device, wherein the additional raw data received from the second remote device has an undefined source type; and
recommend to an administrator of the second remote device that the source type of the additional raw data be set to a name specified by an administrator of the first remote device.
2. The system of claim 1 wherein the instructions, when executed, cause the set of one or more processors to present the generated parser rule to an administrator of the first remote device.
3. The system of claim 2 wherein the instructions, when executed, cause the set of one or more processors to receive a confirmation from the administrator that the generated parser rule is correct.
4. The system of claim 2 wherein the instructions, when executed, cause the set of one or more processors to receive a modified version of the generated parser rule from the administrator.
5. The system of claim 1 wherein generating at least one parser rule includes generating a regular expression.
6. The system of claim 1 wherein the set of rules is included in a library, and wherein storing the generated at least one parser rule comprises including the generated at least one parser rule in the library.
7. The system of claim 1 wherein the instructions, when executed, cause the set of one or more processors to ask an administrator of the first remote device to provide a name of a source of the raw data and associate the name with the generated parser rule.
8. The system of claim 1 wherein the instructions, when executed, cause the set of one or more processors to evaluate the raw data received from the second remote device against the generated parser rule.
9. A method, comprising:
receiving, via a set of one or more interfaces, raw data from a first remote device;
determining, using a set of one or more processors, that the raw data does not, within a predefined confidence measure, conform to any rules included in a set of rules;
performing, using the set of one or more processors, a clustering function on the raw data;
generating, using the set of one or more processors, at least one parser rule based at least in part on the clustering;
storing, using the set of one or more processors, the generated at least one parser rule in a data store;
receiving, via the set of one or more interfaces, additional raw data from a second remote device, wherein the additional raw data received from the second remote device has an undefined source type; and
recommending, using the set of one or more processors, to an administrator of the second remote device that the source type of the additional raw data be set to a name specified by an administrator of the first remote device.
10. The method of claim 9 further comprising presenting, using the set of one or more processors, the generated parser rule to an administrator of the first remote device.
11. The method of claim 10 further comprising receiving, via the set of one or more interfaces, a confirmation from the administrator that the generated parser rule is correct.
12. The method of claim 10 further comprising receiving, via the set of one or more interfaces, a modified version of the generated parser rule from the administrator.
13. The method of claim 9 further comprising asking, using the set of one or more processors, an administrator of the first remote device to provide a name of a source of the raw data and wherein the processor is further configured to associate the name with the generated parser rule.
14. The method of claim 9 further comprising evaluating, using the set of one or more processors, the raw data received from the second remote device against the generated parser rule.
15. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, via a set of one or more interfaces, raw data from a first remote device;
determining, using a set of one or more processors, that the raw data does not, within a predefined confidence measure, conform to any rules included in a set of rules;
performing, using the set of one or more processors, a clustering function on the raw data;
generating, using the set of one or more processors, at least one parser rule based at least in part on the clustering; and
storing, using the set of one or more processors, the generated at least one parser rule in a data store; and
receiving, via the set of one or more interfaces, additional raw data from a second remote device, wherein the additional raw data received from the second remote device has an undefined source type; and
recommending, using the set of one or more processors, to an administrator of the second remote device that the source type of the additional raw data be set to a name specified by an administrator of the first remote device.