Selection of representative data subsets from groups of events
Embodiments are directed towards generating a representative sampling as a subset from a larger dataset that includes unstructured data. A graphical user interface enables a user to provide various data selection parameters, including specifying a data source and one or more subset types desired, including one or more of latest records, earliest records, diverse records, outlier records, and/or random records. Diverse and/or outlier subset types may be obtained by generating clusters from an initial selection of records obtained from the larger dataset. An iteration analysis is performed to determine whether a sufficient number of clusters and/or cluster types have been generated that exceed at least one threshold and when not exceeded, additional clustering is performed on additional records. From the resultant clusters, and/or other subtype results, a subset of records is obtained as the representative sampling subset.
1. A computer-implemented method comprising:
retrieving a plurality of events from a data source in accordance with a selected data subset type of a plurality of defined, user-selectable data subset types;
identifying similarity between two or more of the retrieved events;
determining whether any of the retrieved events form a group, based on the identified similarity;
selecting, based on a determination that two or more of the retrieved events form a group, at least a subset of the retrieved events that form the group, as a representative data subset; and
causing display of the selected events in a user interface as the representative data subset, wherein causing display of the selected events in a user interface as the representative data subset comprises causing display of the selected events in a user interface that enables development of a field extraction rule that specifies how to extract a value for a field from information in one or more events.
2. The method of claim 1 , wherein said determining whether any of the retrieved events form a group comprises applying a clustering algorithm to the retrieved events.
3. The method of claim 1 , further comprising:
receiving, from a user, a selection of a data source type from which to generate the representative data subset.
4. The method of claim 1 , further comprising:
receiving, from a user, a selection of the data subset type, of the plurality of defined data subset types, for identifying an event to include in the representative data subset.
5. The method of claim 1 , further comprising:
receiving, from a user, a selection of a number of desired representative events to be included in the representative data subset.
6. The method of claim 1 , further comprising:
receiving, from a user, selections of:
a data source type from which to generate the representative data subset,
one or a combination of subset types, of a plurality of defined data subset types, for identifying an event to include in the representative data subset, and
a number of desired representative events to be included in the representative data subset.
7. The method of claim 1 , wherein each of the retrieved events includes raw data indicative of performance or activity of one or more components of an information technology environment.
8. The method of claim 1 , wherein the plurality of defined data subset types corresponds to a plurality of subtype processes that include one or more of a diverse event-identification process, an outlier event-identification process, a random event identification process, an earlier event-identification process, or a later event-identification process.
9. The method of claim 1 , wherein determining whether any of the retrieved events form a group is part of applying a clustering algorithm to the plurality of events to form a plurality of clusters;
the method further comprising:
determining that a number of clusters in the plurality of clusters is not of a sufficiently large; and
clustering a larger group of events in the plurality of events than the group of events.
10. The method of claim 1 , wherein each event in the plurality of events is associated with a time stamp.
11. The method of claim 1 , wherein each event in the plurality of events is associated with a time stamp that has been extracted from data in that event.
12. The method of claim 1 , wherein retrieving events from the data source includes using a process to identify outlier events.
13. The method of claim 1 , wherein retrieving events from the data source includes using a process to identify events associated with earliest events in the plurality of events.
14. The method of claim 1 , wherein retrieving events from the data source includes using a process to identify events associated with latest events in the plurality of events.
15. A non-transitory, machine-readable storage medium storing instructions, execution of which in a computer system causes the computer system to perform operations comprising:
retrieving a plurality of events from a data source in accordance with a selected data subset type of a plurality of defined, user-selectable data subset types;
identifying similarity between two or more of the retrieved events;
determining whether any of the retrieved events form a group, based on the identified similarity;
selecting, based on a determination that two or more of the retrieved events form a group, at least a subset of the retrieved events that form the group, as a representative data subset; and
causing display of the selected events in a user interface as the representative data subset, wherein causing display of the selected events in a user interface as the representative data subset comprises causing display of the selected events in a user interface that enables development of a field extraction rule that specifies how to extract a value for a field from information in one or more events.
16. The non-transitory, machine-readable storage medium of claim 15 , further comprising:
receiving, from a user, a selection of a data source type from which to generate the representative data subset; and
receiving, from the user, a selection of the data subset type, of the plurality of defined data subset types, for identifying an event to include in the representative data subset.
17. A computer system comprising:
a data store to store data; and
a processor coupled to the data store and configured to perform or cause operations that include
retrieving a plurality of events from a data source in accordance with a selected data subset type of a plurality of defined, user-selectable data subset types;
identifying similarity between two or more of the retrieved events;
determining whether any of the retrieved events form a group, based on the identified similarity;
selecting, based on a determination that two or more of the retrieved events form a group, at least a subset of the retrieved events that form the group, as a representative data subset; and
causing display of the selected events in a user interface as the representative data subset, wherein causing display of the selected events in a user interface as the representative data subset comprises causing display of the selected events in a user interface that enables development of a field extraction rule that specifies how to extract a value for a field from information in one or more events.