Extraction rule generation using clustering
Determining a set of extraction rules include clustering event segments into at least a first group of event segments, and determining, using first field data in the first group of event segments, a first set of extraction rules for extracting the first field data from each event segment of the first group of event segments. A determination is made that the first set of extraction rules fails to successfully extract all of the first field data. Responsive to the determination, the event segments are re-clustered into at least a second group of event segments and a third group of event segments until a successful set of extraction rules are identified. The successful set of extraction rules are stored in computer memory.
1. A computer implemented method, comprising:
clustering a plurality of event segments into at least a first group of event segments, wherein each event segment of the plurality of event segments corresponds to an individual event in machine data;
determining, using first field data in the first group of event segments, a first set of extraction rules for extracting the first field data in a first plurality of fields from each event segment of the first group of event segments;
making a determination that the first set of extraction rules fail to successfully extract all of the first field data in the first plurality of fields;
re-clustering, responsive to the determination, the plurality of event segments into at least a second group of event segments and a third group of event segments until a successful set of extraction rules are identified,
wherein identifying the successful set of extraction rules comprises:
examining, for the second group of event segments, location, and content of second field data in the second group of event segments to determine a second set of extraction rules that extracts the second field data from a second plurality of fields in the second group, and
examining, for the third group of event segments, location, and content of third field data in the third group of event segments to determine a third set of extraction rules that extracts the third field data from a fourth plurality of fields in the third group; and
storing the successful set of extraction rules in computer memory.
2. The computer implemented method of claim 1 ,
wherein clustering the plurality of event segments into at least the first group of event segments further comprises clustering the plurality of event segments into at least the first group of event segments and a fourth group of event segments; and
wherein the method further comprises determining, using fourth field data in the fourth group of event segments, a fourth set of extraction rules for extracting the fourth field data in a fourth plurality of fields from each event segment of the fourth group of event segments.
3. The computer implemented method of claim 1 , further comprising:
receiving a sample of machine data in a form produced by a data source; and
performing pattern recognition to predict breakpoints in the sample, the breakpoints identifying boundaries between distinct event segments of the plurality of event segments of the sample.
4. The computer implemented method of claim 1 , further comprising:
receiving a sample of machine data in a form produced by a data source; and
comparing patterns in the sample with a plurality of delimiter patterns for a plurality of data source types to predict breakpoints in the sample, the breakpoints identifying boundaries between distinct event segments of the plurality of event segments of the sample.
5. The computer implemented method of claim 1 , further comprising:
parsing each of the plurality of event segments into a plurality of fields.
6. The computer implemented method of claim 1 , wherein the clustering is based at least in part on one or more from among connectivity-based clustering, centroid-based clustering, distribution-based clustering, density-based clustering, canopy clustering, K-means clustering, subspace clustering, and correlation clustering.
7. The computer implemented method of claim 1 , wherein the successful set of extraction rules are stored as an extraction model.
8. The computer implemented method of claim 1 , wherein the successful set of extraction rules are stored as an extraction model of a data sourcetype.
9. The computer implemented method of claim 1 , wherein classifying includes automatically identifying one or more fields in an event segment by matching patterns associated with one or more known fields of a late-binding schema, the known fields having an association with a domain category.
10. The computer implemented method of claim 1 , further comprising:
displaying the successful set of extraction rules in a graphical user interface (GUI), and
storing the successful set of extraction rules responsive to user acceptance via the GUI.
11. The computer implemented method of claim 1 , further comprising:
displaying, in a graphical user interface (GUI), a depiction of an event segment in the plurality of event segments with an extraction rule in the successful set of extraction rules in a graphical user interface (GUI), and
storing the successful set of extraction rules responsive to user acceptance via the GUI.
12. The computer implemented method of claim 1 , further comprising:
displaying, in a graphical user interface (GUI), a depiction of an event segment having one or more field portions color-coded in accordance with a particular set of extraction rules, the event segment in the plurality of event segments, and the particular set of extraction rules in the successful set of extraction rules, and
storing the successful set of extraction rules responsive to user acceptance via the GUI.
13. A system comprising:
a memory; and
a processing device coupled with the memory to perform operations comprising:
clustering a plurality of event segments into at least a first group of event segments, wherein each event segment of the plurality of event segments corresponds to an individual event in machine data;
determining, using first field data in the first group of event segments, a first set of extraction rules for extracting the first field data in a first plurality of fields from each event segment of the first group of event segments;
making a determination that the first set of extraction rules fail to successfully extract all of the first field data in the first plurality of fields;
re-clustering, responsive to the determination, the plurality of event segments into at least a second group of event segments and a third group of event segments until a successful set of extraction rules are identified, wherein identifying the successful set of extraction rules comprises:
examining, for the second group of event segments, location, and content of second field data in the second group of event segments to determine a second set of extraction rules that extracts the second field data from a second plurality of fields in the second group, and
examining, for the third group of event segments, location, and content of third field data in the third group of event segments to determine a third set of extraction rules that extracts the third field data from a fourth plurality of fields in the third group; and
storing the successful set of extraction rules in computer memory.
14. The system of claim 13 , the operations further comprising:
receiving a sample of machine data in a form produced by a data source; and
performing pattern recognition to predict breakpoints in the sample, the breakpoints identifying boundaries between distinct event segments of the plurality of event segments of the sample.
15. The system of claim 13 , the operations further comprising:
receiving a sample of machine data in a form produced by a data source; and
comparing patterns in the sample with a plurality of delimiter patterns for a plurality of data source types to predict breakpoints in the sample, the breakpoints identifying boundaries between distinct event segments of the plurality of event segments of the sample.
16. The system of claim 13 , wherein the clustering is based at least in part on one or more from among connectivity-based clustering, centroid-based clustering, distribution-based clustering, density-based clustering, canopy clustering, K-means clustering, subspace clustering, and correlation clustering.
17. The system of claim 13 , the operations further comprising:
displaying, in a graphical user interface (GUI), a depiction of an event segment having one or more field portions color-coded in accordance with a particular set of extraction rules, the event segment in the plurality of event segments, and the particular set of extraction rules in the successful set of extraction rules, and
storing the successful set of extraction rules responsive to user acceptance via the GUI.
18. A non-transitory computer readable storage medium encoding instructions thereon that, in response to execution by one or more processing devices, cause the one or more processing devices to perform operations comprising:
clustering a plurality of event segments into at least a first group of event segments, wherein each event segment of the plurality of event segments corresponds to an individual event in machine data;
determining, using first field data in the first group of event segments, a first set of extraction rules for extracting the first field data in a first plurality of fields from each event segment of the first group of event segments;
making a determination that the first set of extraction rules fail to successfully extract all of the first field data in the first plurality of fields;
re-clustering, responsive to the determination, the plurality of event segments into at least a second group of event segments and a third group of event segments until a successful set of extraction rules are identified,
wherein identifying the successful set of extraction rules comprises:
examining, for the second group of event segments, location, and content of second field data in the second group of event segments to determine a second set of extraction rules that extracts the second field data from a second plurality of fields in the second group, and
examining, for the third group of event segments, location, and content of third field data in the third group of event segments to determine a third set of extraction rules that extracts the third field data from a fourth plurality of fields in the third group; and
storing the successful set of extraction rules in computer memory.
19. The non-transitory computer readable storage medium of claim 18 , the operations further comprising:
receiving a sample of machine data in a form produced by a data source; and
performing pattern recognition to predict breakpoints in the sample, the breakpoints identifying boundaries between distinct event segments of the plurality of event segments of the sample.
20. The non-transitory computer readable storage medium of claim 18 , the operations further comprising:
receiving a sample of machine data in a form produced by a data source; and
comparing patterns in the sample with a plurality of delimiter patterns for a plurality of data source types to predict breakpoints in the sample, the breakpoints identifying boundaries between distinct event segments of the plurality of event segments of the sample.