Index time, delimiter based extractions and previewing for use in indexing
A graphical user interface allows a customer to specify delimiters and/or patterns that occur in event data and indicate the presence of a particular field. The graphical user interface applies a customer's delimiter specifications directly to event data and displays the resulting event data in real time. Delimiter specifications may be saved as configuration settings and systems in a distributed setting may use the delimiter specifications to extract field values as the systems process raw data into event data. Extracted field values are used to accelerate search queries that a system receives.
1. A method, comprising:
retrieving one or more events;
causing display of a graphical user interface that displays one or more field delimiter options specifying one or more delimiters that indicate a boundary of a field value;
in response to a selection of a field delimiter option of the one or more field delimiter options, parsing at least one of the one or more events to identify field values using the selected field delimiter option;
causing display of the field values identified in the at least one of the one or more events as the parsing occurs;
storing the selected field delimiter option and one or more associated field names in a configuration file, wherein the configuration file specifies configuration parameters for field extraction during raw data indexing;
receiving raw data from a data source;
parsing the raw data into a plurality of timestamped events, each timestamped event in the plurality of timestamped events comprising at least a portion of the parsed raw data;
concurrent with parsing the raw data into a plurality of timestamped events, identifying a particular field in the timestamped events using the selected field delimiter obtained from the configuration file that is associated with the particular field; and
storing a field value pair for each unique value extracted from the particular field in the timestamped events along with an associated field name obtained from the configuration file on at least one storage device.
2. The method of claim 1 , wherein the specified field delimiter is a character.
3. The method of claim 1 , wherein the specified field delimiter is a pattern of characters.
4. The method of claim 1 , wherein one or more associated field names is specified along with the specified field delimiter.
5. The method of claim 1 , wherein
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs.
6. The method of claim 1 , further comprising:
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame; and
associating a set of stored field value pairs with a group of events.
7. The method of claim 1 , further comprising:
indexing the plurality of timestamped events derived from the raw data;
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame; and
associating a set of stored field value pairs with a group of events.
8. The method of claim 1 , further comprising:
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs;
receiving a search query;
determining that the search query refers to a field value pair;
searching the stored field value pairs for a field value pair that satisfies the search query;
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
9. The method of claim 1 , further comprising:
indexing a plurality of timestamped events derived from the raw data;
storing associated event identification information that identifies each event where the field value pair occurs;
receiving a search query;
determining that the search query refers to a field value pair;
searching the stored field value pairs for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
10. The method of claim 1 , further comprising:
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs;
associating a set of stored field value pairs with a group of events;
receiving a search query;
determining that the search query refers to a field value pair and the group of events;
searching the stored field value pairs associated with the group of events for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
11. The method of claim 1 , further comprising:
indexing a plurality of timestamped events derived from the raw data;
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs;
associating a set of stored field value pairs with a group of events;
receiving a search query;
determining that the search query refers to a field value pair and the group of events;
searching the stored field value pairs associated with the group of events for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
12. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of:
retrieving one or more events;
causing display of a graphical user interface that displays one or more field delimiter options specifying one or more delimiters that indicate a boundary of a field value;
in response to a selection of a field delimiter option of the one or more field delimiter options, parsing at least one of the one or more events to identify field values using the selected field delimiter option;
causing display of the field values identified in the at least one of the one or more events as the parsing occurs;
storing the selected field delimiter option and one or more associated field names in a configuration file, wherein the configuration file specifies configuration parameters for field extraction during raw data indexing;
receiving raw data from a data source;
parsing the raw data into a plurality of timestamped events, each timestamped event in the plurality of timestamped events comprising at least a portion of the parsed raw data;
concurrent with parsing the raw data into a plurality of timestamped events, identifying a particular field in the timestamped events using the selected field delimiter obtained from the configuration file that is associated with the particular field; and
storing a field value pair for each unique value extracted from the particular field in the timestamped events along with an associated field name obtained from the configuration file on at least one storage device.
13. The non-transitory computer readable storage medium of claim 12 , wherein the specified field delimiter is a character.
14. The non-transitory computer readable storage medium claim 12 , wherein the specified field delimiter is a pattern of characters.
15. The non-transitory computer readable storage medium of claim 12 , further comprising:
indexing a plurality of timestamped events derived from the raw data;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs.
16. The non-transitory computer readable storage medium of claim 12 , further comprising:
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs; and
associating a set of stored field value pairs with a group of events.
17. The non-transitory computer readable storage medium of claim 12 , further comprising:
indexing a plurality of timestamped events derived from the raw data;
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs; and
associating a set of stored field value pairs with a group of events.
18. The non-transitory computer readable storage medium of claim 12 , further comprising:
storing associated event identification information that identifies each event where the field value pair occurs;
receiving a search query;
determining that the search query refers to a field value pair;
searching the stored field value pairs for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
19. The non-transitory computer readable storage medium claim 12 , further comprising:
indexing a plurality of timestamped events derived from the raw data;
storing associated event identification information that identifies each event where the field value pair occurs;
receiving a search query;
determining that the search query refers to a field value pair;
searching the stored field value pairs for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
20. A system including one or more processors coupled to memory, the memory loaded with computer instructions that, when executed on the processors, implement actions comprising:
retrieving one or more events;
causing display of a graphical user interface that displays one or more field delimiter options specifying one or more delimiters that indicate a boundary of a field value;
in response to a selection of a field delimiter option of the one or more field delimiter options, parsing at least one of the one or more events to identify field values using the selected field delimiter option;
causing display of the field values identified in the at least one of the one or more events as the parsing occurs;
storing the selected field delimiter option and one or more associated field names in a configuration file, wherein the configuration file specifies configuration parameters for field extraction during raw data indexing;
receiving raw data from a data source;
parsing the raw data into a plurality of timestamped events, each timestamped event in the plurality of timestamped events comprising at least a portion of the parsed raw data;
concurrent with parsing the raw data into a plurality of timestamped events, identifying a particular field in the timestamped events using the selected field delimiter obtained from the configuration file that is associated with the particular field; and
storing a field value pair for each unique value extracted from the particular field in the timestamped events along with an associated field name obtained from the configuration file on at least one storage device.
21. The system of claim 20 , wherein the specified field delimiter is a character.
22. The system of claim 20 , wherein the specified field delimiter is a pattern of characters.
23. The system of claim 20 , wherein one or more associated field names is specified along with the specified field delimiter.
24. The system of claim 20 , further comprising:
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs.
25. The system of claim 20 , further comprising:
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs; and
associating a set of stored field value pairs with a group of events.
26. The system of claim 20 , further comprising:
indexing the plurality of timestamped events derived from the raw data;
organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs; and
associating a set of stored field value pairs with a group of events.
27. The system of claim 20 , further comprising:
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs;
receiving a search query;
determining that the search query refers to a field value pair;
searching the stored field value pairs for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.
28. The system of claim 20 , further comprising:
indexing a plurality of timestamped events derived from the raw data;
for each field value pair, storing associated event identification information that identifies each event where the field value pair occurs;
receiving a search query;
determining that the search query refers to a field value pair;
searching the stored field value pairs for a field value pair that satisfies the search query; and
in response to finding a field value pair that satisfies the search query, returning search results comprising information related to one or more events identified in the event identification information associated with the field value pair that satisfies the search query.