Automatic parser generation
Automatically generating a parser is disclosed. Raw data is received from a first remote device. A determination that the raw data does not, within a predefined confidence measure, conform to any rules included in a set of rules is made. A clustering function is performed on the raw data. At least one parser rule is generated based on the clustering.
1. A system, comprising:
a processor configured to:
receive raw data generated at a first remote device;
calculate at least one confidence measure indicating a level of correspondence between the raw data and at least one parser rule included in an existing set of parser rules;
determine that the raw data does not conform to any parser rules included in the existing set of parser rules based at least in part on a determination that the at least one confidence measure does not meet a predefined threshold;
in response to the determination that the raw data does not conform to any parser rules included in the existing set, obtain structure information associated with the raw data, wherein obtaining the structure information includes at least one of:
determining a first segment of the raw data that is populated by a repeating static value; and
determining a second segment of the raw data that is populated by variable values included in a set comprising a plurality of such variable values;
automatically generate at least one new parser rule based at least in part on the structure information;
parse at least a portion of the raw data using the at least one new parser rule;
receive additional raw data generated at a second remote device, wherein the additional raw data generated at the second remote device has an unspecified source; and
recommend to an administrator of the second remote device that the source of the additional raw data be set to a name provided by an administrator of the first remote device; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 wherein the processor is further configured to present the generated parser rule to the administrator of the first remote device.
3. The system of claim 2 wherein the processor is further configured to receive a confirmation from the administrator of the first remote device that the generated parser rule is correct.
4. The system of claim 2 wherein the processor is further configured to receive a modified version of the generated parser rule from the administrator of the first remote device.
5. The system of claim 1 wherein generating at least one parser rule includes generating a regular expression.
6. The system of claim 1 wherein the processor is further configured to store the generated parser rule.
7. The system of claim 1 wherein the processor is further configured to ask the administrator of the first remote device to provide a name of a source of the raw data and wherein the processor is further configured to associate the name with the generated parser rule.
8. The system of claim 1 wherein the processor is further configured to evaluate the additional raw data generated at the second remote device against the generated parser rule.
9. A method, comprising:
receiving raw data generated at a first remote device;
calculating at least one confidence measure indicating a level of correspondence between the raw data and at least one parser rule included in an existing set of parser rules;
determining that the raw data does not conform to any parser rules included in the existing set of parser rules based at least in part on a determination that the at least one confidence measure does not meet a predefined threshold;
in response to the determination that the raw data does not conform to any parser rules included in the existing set, obtaining structure information associated with the raw data, wherein obtaining the structure information includes at least one of:
determining a first segment of the raw data that is populated by a repeating static value; and
determining a second segment of the raw data that is populated by variable values included in a set comprising a plurality of such variable values;
automatically generating at least one new parser rule based at least in part on the structure information;
parsing at least a portion of the raw data using the at least one new parser rule;
receiving additional raw data generated at a second remote device, wherein the additional raw data generated at the second remote device has an unspecified source; and
recommending to an administrator of the second remote device that the source of the additional raw data be set to a name provided by an administrator of the first remote device.
10. The method of claim 9 further comprising presenting the generated parser rule to the administrator of the first remote device.
11. The method of claim 10 further comprising receiving a confirmation from the administrator of the first remote device that the generated parser rule is correct.
12. The method of claim 10 further comprising receiving a modified version of the generated parser rule from the administrator of the first remote device.
13. The method of claim 9 further comprising asking the administrator of the first remote device to provide a name of a source of the raw data and wherein the processor is further configured to associate the name with the generated parser rule.
14. The method of claim 9 further comprising evaluating the additional raw data generated at the second remote device against the generated parser rule.
15. The method of claim 9 wherein generating at least one parser rule includes generating a regular expression.
16. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving raw data generated at a first remote device;
calculating at least one confidence measure indicating a level of correspondence between the raw data and at least one parser rule included in an existing set of parser rules;
determining that the raw data does not conform to any parser rules included in the existing set of parser rules based at least in part on a determination that the at least one confidence measure does not meet a predefined threshold;
in response to the determination that the raw data does not conform to any parser rules included in the existing set, obtaining structure information associated with the raw data, wherein obtaining the structure information includes at least one of:
determining a first segment of the raw data that is populated by a repeating static value; and
determining a second segment of the raw data that is populated by variable values included in a set comprising a plurality of such variable values;
automatically generating at least one new parser rule based at least in part on the structure information;
parsing at least a portion of the raw data using the at least one new parser rule;
receiving additional raw data generated at a second remote device, wherein the additional raw data generated at the second remote device has an unspecified source; and
recommending to an administrator of the second remote device that the source of the additional raw data be set to a name provided by an administrator of the first remote device.
17. The computer program product of claim 16 further comprising computer instructions for presenting the generated parser rule to the administrator of the first remote device.
18. The computer program product of claim 17 further comprising computer instructions for receiving a confirmation from the administrator of the first remote device that the generated parser rule is correct.
19. The computer program product of claim 17 further comprising computer instructions for receiving a modified version of the generated parser rule from the administrator of the first remote device.
20. The computer program product of claim 16 further comprising computer instructions for asking an administrator of the first remote device to provide a name of a source of the raw data and wherein the processor is further configured to associate the name with the generated parser rule.