Sampling of events to use for developing a field-extraction rule for a field to use in event searching
Embodiments are directed towards generating a representative sampling as a subset from a larger dataset that includes unstructured data. A graphical user interface enables a user to provide various data selection parameters, including specifying a data source and one or more subset types desired, including one or more of latest records, earliest records, diverse records, outlier records, and/or random records. Diverse and/or outlier subset types may be obtained by generating clusters from an initial selection of records obtained from the larger dataset. An iteration analysis is performed to determine whether a sufficient number of clusters and/or cluster types have been generated that exceed at least one threshold and when not exceeded, additional clustering is performed on additional records. From the resultant clusters, and/or other subtype results, a subset of records is obtained as the representative sampling subset.
1. A computer-implemented method, comprising:
receiving machine data at a computing device;
generating a plurality of events, wherein each event in the plurality of events includes a portion of the machine data;
associating a time with each event in the plurality of events, the time for each event extracted from the machine data included in that event;
storing the plurality of events in a data store such that they are searchable at least by their associated times;
receiving from a user a selection of one or more event selection parameters;
wherein each event selection parameter corresponds to a distinct process for identifying events for inclusion in a set;
wherein the one or more event selection parameters selected by the user include at least one of diverse, outlier, random, earliest, and latest event selection processes;
for each of the received one or more event selection parameters, identifying events for inclusion in the set using the corresponding distinct processes; and
displaying one or more events in the set of events in a graphical user interface that enables development of a field-extraction rule that specifies how to extract, from the machine data included in each of the one or more events, a value for a field that is defined for each of the one or more events, wherein each of the one or more events is searchable using the field.
2. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify diverse events.
3. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify outlier events.
4. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify events associated with earliest events in the plurality of events.
5. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify events associated with latest events in the plurality of events.
6. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process randomly to identify events in the plurality of events.
7. The method of claim 1 , wherein identifying events for inclusion in the set includes using a combination of two or more processes selected from a process to identify diverse events, a process to identify outlier events, a process to identify earliest events in the plurality of events, a process to identify latest events in the plurality of events, and a process randomly to identify events in the plurality of events.
8. The method of claim 1 , identifying events for inclusion in the set includes using a process to identify diverse events, and wherein the process to identify diverse events includes:
performing a clustering algorithm on a group of events from the plurality of events to form a plurality of clusters, the clustering algorithm placing two events into a same cluster based on similarities in the machine data included in each of the two events; and
selecting events from one or more most populous clusters in the plurality of clusters.
9. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify outlier events, and wherein the process to identify outlier events includes:
performing a clustering algorithm on a group of events from the plurality of events to form a plurality of clusters, the clustering algorithm placing two events into a same cluster based on similarities in the machine data included in each of the two events; and
selecting events from one or more least populous clusters in the plurality of clusters.
10. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify diverse events or using a process to identify outlier events, and wherein the process includes:
clustering a group of events in the plurality of events to form a plurality of clusters;
determining that a number of clusters in the plurality of clusters is not big enough; and
clustering a larger group of events in the plurality of events than the group of events.
11. The method of claim 1 , wherein identifying events for inclusion in the set includes using a process to identify diverse events or using a process to identify outlier events, and wherein the process includes:
clustering a group of events in the plurality of events to form a plurality of clusters;
determining that a number of events in one of the clusters in the plurality of clusters is not big enough; and
clustering a larger group of events in the plurality of events than the group of events.
12. The method of claim 1 , wherein receiving from the user the selection of one or more event selection parameters comprises causing display of a graphical interface that provides a plurality of event selection parameters that can be selected, and wherein the selected event selection parameters are received through the graphical interface.
13. A non-transitory, computer-readable medium having computer executable instructions for performing the method of claim 1 .
14. A computer system with one or more processors adapted to perform the method of claim 1 .
15. A computer-implemented method, comprising:
receiving machine data at a computing device;
generating a plurality of events, wherein each event in the plurality of events includes a portion of the machine data;
associating a time with each event in the plurality of events, the time for each event extracted from the machine data included in that event;
storing the plurality of events in a data store such that they are searchable at least by their associated times;
selecting a set of events from the plurality of events using a process to identify diverse events, the process including:
performing a clustering algorithm on a group of events from the plurality of events to form a plurality of clusters, the clustering algorithm placing two events into a same cluster based on similarities in the machine data included in each of the two events; and
selecting events from one or more most populous clusters in the plurality of clusters; and
displaying one or more events in the set of events in a graphical user interface that enables development of a field-extraction rule that specifies how to extract, from the machine data included in each of the one or more events, a value for a field that is defined for each of the one or more events, wherein each of the one or more events is searchable using the field.
16. A non-transitory, computer-readable medium having computer executable instructions for performing the method of claim 15 .
17. A computer system with one or more processors adapted to perform the method of claim 15 .
18. A computer-implemented method, comprising:
receiving machine data at a computing device;
generating a plurality of events, wherein each event in the plurality of events includes a portion of the machine data;
associating a time with each event in the plurality of events, the time for each event extracted from the machine data included in that event;
storing the plurality of events in a data store such that they are searchable at least by their associated times;
selecting a set of events from the plurality of events using a process to identify outlier events, the process including:
performing a clustering algorithm on a group of events from the plurality of events to form a plurality of clusters, the clustering algorithm placing two events into a same cluster based on similarities in the machine data included in each of the two events; and
selecting events from one or more least populous clusters in the plurality of clusters; and
displaying one or more events in the set of events in a graphical user interface that enables development of a field-extraction rule that specifies how to extract, from the machine data included in each of the one or more events, a value for a field that is defined for each of the one or more events, wherein each of the one or more events is searchable using the field.
19. A non-transitory, computer-readable medium having computer executable instructions for performing the method of claim 18 .
20. A computer system with one or more processors adapted to perform the method of claim 18 .