Data categorization using inverted indexes
Systems and methods are disclosed involving user interface (UI) search tools for locating data, including tools for summarizing indexed raw machine data that organize and present results to enable expansion and exploration of initial summarizations. The initial summarizations may be explored and refined to help users determine how to identify and best focus a search on data subsets of greater interest.
1. A method, comprising:
maintaining, by a processing device, a plurality of inverted indexes, each inverted index of inverted indexes comprising a plurality of entries, each entry of the entries comprising:
a token or a field-value pair, and
one or more event references, each event reference of the one or more event references indicative of an event that is associated with the token or a field-value corresponding to the field-value pair;
identifying, by the processing device, a set of the event references from the plurality of inverted indexes;
categorizing, by the processing device using a plurality of field-value pairs in at least a portion of the plurality of inverted indexes, the set of event references according to categorization criteria, wherein the categorization criteria identifies one or more fields to categorize the set of event references, wherein categorization criteria-value pairs correspond to unique combinations of the one or more fields and field values of the one or more fields; and
in response to a request received from a device:
generating one or more groupings based at least in part on the categorizing the set of event references, wherein a given grouping of the one or more groupings includes a group of the event references corresponding to a group of events that have the same field-value for a field of the one or more fields; and
causing display, via a graphical user interface, of the one or more groupings.
2. The method of claim 1 , wherein the plurality of inverted indexes have information regarding at least one of a partition associated with events or an origin of the events.
3. The method of claim 1 , wherein the plurality of inverted indexes have information regarding at least one of a partition associated with events or an origin of the events, each event of the events comprising a portion of raw machine data associated with a time stamp.
4. The method of claim 1 , wherein the plurality of inverted indexes have information regarding at least one of a partition associated with events or an origin of the events, each event of the events comprising a portion of raw machine data associated with a time stamp, wherein the raw machine data is generated by one or more components in an information technology environment.
5. The method of claim 1 , wherein causing display of the one or more groupings is based on at least one of index, host, source, or source type associated with the one or more groupings.
6. The method of claim 1 , wherein the categorization criteria comprises at least one of index, host, source, or source type.
7. The method of claim 1 , further comprising:
obtaining filter criteria specifying data for categorization; and
identifying a subset of the plurality of inverted indexes based at least in part on the filter criteria, wherein identifying the set of event references comprises identifying the set of event references from the subset of the plurality of inverted indexes based at least in part on the filter criteria.
8. The method of claim 1 , wherein the one or more groupings are displayed with a count corresponding to a number of the group of events associated with matching categorization criteria-value pairs.
9. The method of claim 1 , further comprising:
based on an interaction with a display object associated with the given grouping, sampling the group of events; and
generating a summary of the sampling.
10. The method of claim 1 , wherein each grouping of the one or more groupings corresponds to a particular combination of the one or more fields and the field values.
11. The method of claim 1 , wherein the one or more groupings are displayed based on display criteria, including at least one of a sort order or a split by order.
12. The method of claim 1 , wherein each inverted index of the plurality of inverted indexes further comprises an event reference array comprising a timestamp and a location identifier for each event reference that is located in the respective inverted index.
13. The method of claim 1 , wherein the plurality of entries comprises:
one or more token entries, each token entry comprising:
a particular token, and
at least one event reference indicative of an event that includes the token, and
one or more field-value pair entries, each field-value pair entry comprising:
a particular field-value pair, and
at least one event reference indicative of an event that includes a particular field-value corresponding to the particular field-value pair,
wherein identifying the set of event references comprises identifying at least one event reference located in each token entry and each field-value pair entry identified by filter criteria.
14. The method of claim 1 , further comprising:
obtaining filter criteria specifying data for categorization, wherein the filter criteria comprises a time range; and
identifying a subset of the plurality of inverted indexes based at least in part on the filter criteria, wherein identifying the set of event references comprises identifying the set of event references from the subset of the plurality of inverted indexes based at least in part on referring to an event reference array to determine event references that correspond to events satisfying the time range.
15. The method of claim 1 , wherein the categorization criteria comprises an index and wherein the one or more groupings comprise at least one grouping for each distinct index.
16. The method of claim 1 , wherein the categorization criteria comprises a source type and wherein the one or more groupings comprise at least one grouping for each distinct source type field-value pair found in the plurality of entries that include an event reference of the set of event references.
17. The method of claim 1 , wherein the categorization criteria comprises a host and wherein the one or more groupings comprise at least one grouping for each distinct host field-value pair found in the plurality of entries that include an event reference of the set of event references.
18. The method of claim 1 , wherein the categorization criteria comprises a source and wherein the one or more groupings comprise at least one grouping for each distinct source field-value pair found in the plurality of entries that include an event reference of the set of event references.
19. A computing system, comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
maintain a plurality of inverted indexes, each inverted index of inverted indexes comprising a plurality of entries, each entry of the entries comprising:
a token or a field-value pair, and
one or more event references, each event reference of the one or more event references indicative of an event that is associated with the token or a field-value corresponding to the field-value pair;
identify a set of the event references from the plurality of inverted indexes;
categorize, using a plurality of field-value pairs in at least a portion of the plurality of inverted indexes, the set of event references according to categorization criteria, wherein the categorization criteria identifies one or more fields to categorize the set of event references, wherein categorization criteria-value pairs correspond to unique combinations of the one or more fields and field values of the one or more fields; and
in response to a request received from a device:
generate one or more groupings based at least in part on the categorizing the set of event references, wherein a given grouping of the one or more groupings includes a group of the event references corresponding to a group of events that have the same field-value for a field of the one or more fields; and
cause display, via a graphical user interface, of the one or more groupings.
20. Non-transitory computer readable media comprising computer-executable instructions that, when executed by one or more processing devices, cause the one or more processing devices to:
maintain a plurality of inverted indexes, each inverted index of inverted indexes comprising a plurality of entries, each entry of the entries comprising:
a token or a field-value pair, and
one or more event references, each event reference of the one or more event references indicative of an event that is associated with the token or a field-value corresponding to the field-value pair;
identify a set of the event references from the plurality of inverted indexes;
categorize, using a plurality of field-value pairs in at least a portion of the plurality of inverted indexes, the set of event references according to categorization criteria, wherein the categorization criteria identifies one or more fields to categorize the set of event references, wherein categorization criteria-value pairs correspond to unique combinations of the one or more fields and field values of the one or more fields; and
in response to a request received from a device:
generate one or more groupings based at least in part on the categorizing the set of event references, wherein a given grouping of the one or more groupings includes a group of the event references corresponding to a group of events that have the same field-value for a field of the one or more fields; and
cause display, via a graphical user interface, of the one or more groupings.