Optimizing index file sizes based on indexed data storage conditions
Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.
1. A method, comprising:
receiving raw machine data at an indexer, the raw machine data generated by at least one component in an information technology environment and reflecting activity in the information technology environment;
processing, by the indexer, the raw machine data to generate one or more sets of events and an index file associated with at least one set of events of the one or more sets of events, the index file including a keyword portion associating a plurality of keywords with location references to events of the at least one set of events, wherein a keyword of the plurality of keywords is associated with at least one particular location reference to an event of the at least one set of events that includes the keyword;
storing, by the indexer, the at least one set of events and the index file in one or more data stores; and
based at least in part on one or more attributes of the at least one set of events satisfying one or more index size optimization conditions, removing at least a part of the keyword portion from the index file to reduce an amount of storage space used by the index file in the one or more data stores.
2. The method of claim 1 , wherein all of the keyword portion is removed from the index file in response to determining that the one or more index size optimization conditions are met.
3. The method of claim 1 , wherein the index file further includes a metadata portion, the metadata portion including data indicating a range of time associated with the at least one set of events.
4. The method of claim 1 , wherein the index file further includes a metadata portion, the metadata portion including data indicating, for each event of the at least one set of events, an event host value indicating a network host from which a respective event originated, an event source value indicating a data input from which the respective event originated, and an event source type value indicating a format of the data input from which the respective event originated.
5. The method of claim 1 , wherein the one or more index size optimization conditions includes an amount of time elapsed since the data was indexed by the indexer.
6. The method of claim 1 , wherein the one or more index size optimization conditions includes an amount of time elapsed since occurrence of one or more events of the at least one set of events.
7. The method of claim 1 , wherein the one or more index size optimization conditions includes a frequency with which the at least one set of events has been accessed over a period of time.
8. The method of claim 1 , wherein the one or more index size optimization conditions includes an amount of storage space available at a storage device upon which the at least one set of events is stored.
9. The method of claim 1 , wherein the one or more index size optimization conditions includes a type of storage device upon which the at least one set of events is stored.
10. The method of claim 1 , wherein the one or more index size optimization conditions includes an operational state assigned to the at least one set of events.
11. The method of claim 1 , further comprising:
subsequent to removing at least a part of the keyword portion of the index file, receiving a search request including at least one keyword;
performing a sequential scan of the at least one set of events to locate one or more events containing the at least one keyword; and
returning the one or more events as a set of search results.
12. The method of claim 1 , wherein removing at least a part of the keyword portion from the index file includes removing a subset of the plurality of keywords based on a frequency of use of each keyword of the subset.
13. The method of claim 1 , wherein the keyword portion includes a first keyword portion and a second keyword portion, the first keyword portion including a selected set of frequently used keywords, the second keyword portion including other keywords; and
wherein removing at least a part of the keyword portion includes removing the second keyword portion and retaining the first keyword portion.
14. The method of claim 1 , wherein removing at least a part of the keyword portion from the index file includes modifying at least one location reference of the location references to refer to multiple events.
15. The method of claim 1 , wherein removing at least a part of the keyword portion from the index file includes modifying at least one location reference of the location references to refer to multiple events, the method further comprising:
subsequent to removing at least a part of the keyword portion from the index file, receiving a search query including one or more keywords;
searching the index file to identify one or more blocks of events that include the one or more keywords;
performing a sequential scan of the one or more blocks of events to identify one or more individual events that contain at least one of the one or more keywords; and
returning the one or more individual events as a set of search results.
16. The method of claim 1 , further comprising:
subsequent to removing at least a part of the keyword portion from the index file, receiving a search query for data stored in the at least one set of events; and
causing display of an alert indicating that the search query is using a modified index file.
17. The method of claim 1 , further comprising, subsequent to removing at least a part of the keyword portion from the index file, recreating the at least a part of the keyword portion of the index file.
18. The method of claim 1 , further comprising:
subsequent to removing the at least a part of the keyword portion from the index file, determining, by the indexer, that one or more attributes of the at least one set of events meet one or more second index size optimization conditions; and
in response to determining that the one or more attributes of the at least one set of events meet the one or more second index size optimization conditions, recreating the at least a part of the keyword portion of the index file.
19. The method of claim 1 , further comprising recreating the at least a part of the keyword portion of the index file in response to receiving user input requesting to recreate the at least a part of the keyword portion.
20. The method of claim 1 , wherein the indexer is associated with an indexer cluster, the indexer cluster comprising a plurality of peer indexers and a master node, the method further comprising;
in response to determining that the one or more index size optimization conditions are met, sending, to the master node, a request to obtain a lock on the at least one set of events;
in response to receiving the lock on the at least one set of events, removing at least a part of the keyword portion from the index file; and
sending, to the master node, a request to release the lock on the at least one set of events.
21. The method of claim 1 , wherein the indexer is associated with an indexer cluster, the indexer cluster comprising a plurality of peer indexers and a master node, the method further comprising in response to removing at least a part of the keyword portion from the index file, sending a notification to the master node indicating that the indexer removed at least a part of the keyword portion from the index file.
22. Non-transitory, computer readable storage media, storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to:
receive raw machine data, the raw machine data generated by at least one component in an information technology environment and reflecting activity in the information technology environment;
process the raw machine data to generate one or more sets of events and an index file associated with at least one set of events of the one or more sets of events, the index file including a keyword portion associating a plurality of keywords with location references to events of the at least one set of events, wherein a keyword of the plurality of keywords is associated with at least one particular location reference to an event of the at least one set of events that includes the keyword;
store the at least one set of events and the index file in one or more data stores; and
based at least in part on a determination that one or more attributes of the at least one set of events satisfies one or more index size optimization conditions, remove at least a part of the keyword portion from the index file to reduce an amount of storage space used by the index file in the one or more data stores.
23. A computing system, comprising:
one or more processing devices coupled to memory and configured to:
receive raw machine data, the raw machine data generated by at least one component in an information technology environment and reflecting activity in the information technology environment;
process the raw machine data to generate one or more sets of events and an index file associated with at least one set of events of the one or more sets of events, the index file including a keyword portion associating a plurality of keywords with location references to events of the at least one set of events, wherein a keyword of the plurality of keywords is associated with at least one particular location reference to an event of the at least one set of events that includes the keyword;
store the at least one set of events and the index file in one or more data stores; and
based at least in part on a determination that one or more attributes of the at least one set of events satisfies one or more index size optimization conditions, remove at least a part of the keyword portion from the index file to reduce an amount of storage space used by the index file in the one or more data stores.