Managing, storing, and caching query results and partial query results for combination with additional query results
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses one or more containerized search nodes to execute the query and stores the results in a data store for combination with additional query results.
1. A method implemented by a data intake and query system comprising one or more computers, the method comprising:
receiving, at a data intake and query system, a first query identifying a first set of data to be processed and a manner of processing the first set of data;
identifying a plurality of time series buckets that stores the first set of data based at least in part on a respective identifier for each time series bucket of the plurality of time series buckets, wherein the plurality of time series buckets is located in a remote shared storage system, and wherein each time series bucket of the plurality of time series buckets stores raw machine data associated with a timestamp;
identifying a first plurality of containerized search nodes from a group of containerized search nodes to execute the first query, wherein the group of containerized search nodes are implemented as distinct containers in a shared computing resource environment and wherein each time series bucket of the plurality of time series buckets is accessible by any one of the group of containerized search nodes;
executing, by the data intake and query system, the first query using the first plurality of containerized search nodes;
storing results of the first query in a data store for combination with additional query results, wherein the data store is accessible by any one of the group of containerized search nodes;
receiving a second query identifying a second set of data to be processed and a manner of processing the second set of data;
identifying a second plurality of containerized search nodes from the group of containerized search nodes to execute the second query; and
executing the second query using the second plurality of containerized search nodes, wherein the executing the second query comprises combining at least a portion of results of the second query with at least a portion of the results of the first query.
2. The method of claim 1 , wherein the respective identifier for each time series bucket of the plurality of time series buckets is a respective time range for each time series bucket of the plurality of time series buckets.
3. The method of claim 1 ,
wherein the second set of data includes the results of the first query and a third set of data;
wherein executing the second query comprises:
obtaining the at least a portion of the results of the first query from the data store, and
obtaining at least a portion of results of the results of the second query, the at least a portion of results of the second query corresponding to the third set of data;
processing the results of the first query and the partial results; and
storing results of processing the results of the first query and the partial results in the data store as results of the second query.
4. The method of claim 1 ,
wherein the second set of data includes the results of the first query and a third set of data;
wherein executing the second query comprises:
obtaining first partial results of the second query from the data store, the first partial results corresponding to the at least a portion of the results of the first query,
obtaining second partial results of the second query based at least in part on processing the second partial results corresponding to the at least a portion of the results of the second query; and
storing results of the second query in the data store, wherein the results of the second query are based at least in part on a combination of the first partial results and the second partial results.
5. The method of claim 1 , wherein the raw machine data reflects activity in an information processing environment and the raw machine data is produced by a component of the information processing environment.
6. The method of claim 1 , wherein the plurality of time series buckets are field-searchable time series buckets.
7. The method of claim 1 , wherein said executing the first query comprises searching the plurality of time series buckets using a late binding schema.
8. The method of claim 1 , wherein each time series bucket of the plurality of time series buckets comprises a plurality of events and an inverted index corresponding to the plurality of events.
9. The method of claim 1 , further comprising consulting a search node catalog that includes a list of instantiated containerized search nodes of the data intake and query system to identify the first plurality of containerized search nodes.
10. The method of claim 1 , wherein the plurality of time series buckets include raw machine data that satisfies a time range identified by the first query.
11. The method of claim 1 , wherein the plurality of time series buckets include raw machine data associated with a partition identified by the first query.
12. The method of claim 1 , wherein the plurality of time series buckets include raw machine data associated with a tenant identified by the first query.
13. The method of claim 1 , further comprising:
monitoring a status of the first plurality of containerized search nodes; and
based at least in part on said monitoring, updating a search node catalog with the status of the first plurality of containerized search nodes.
14. The method of claim 1 , further comprising:
monitoring a status of the first plurality of containerized search nodes; and
based at least in part on said monitoring, instantiating an additional containerized search node.
15. The method of claim 1 , wherein the remote shared storage system is cloud storage.
16. The method of claim 1 , wherein the remote shared storage system is S3 storage.
17. The method of claim 1 , wherein multiple copies of the plurality of time series buckets are stored in the remote shared storage system.
18. The method of claim 1 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog.
19. The method of claim 1 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog that further includes location information for each time series bucket of the plurality of time series buckets.
20. The method of claim 1 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog that further includes a time range associated with, a tenant associated with, a partition associated with, and location information for, each time series bucket of the plurality of time series buckets.
21. The method of claim 1 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog that is updated over time by an indexing system as buckets are stored in the remote shared storage system by the indexing system.
22. The method of claim 1 , further comprising assigning a containerized search node of the first plurality of containerized search nodes to search a subset of the plurality of time series buckets based at least in part on a search node mapping policy.
23. The method of claim 1 , further comprising assigning a containerized search node of the first plurality of containerized search nodes to search a subset of buckets of the plurality of time series buckets based at least in part on a hash of identifier in the data store catalog of each bucket of the subset of buckets.
24. The method of claim 1 , further comprising assigning a containerized search node of the first plurality of containerized search nodes to search a bucket of the plurality of time series buckets based at least in part on a determination that the containerized search node includes a copy of the bucket on a local or shared data store.
25. The method of claim 1 , further comprising assigning a containerized search node of the first plurality of containerized search nodes to search a bucket of the plurality of time series buckets based at least in part on a determination that the containerized search node was assigned to search the bucket as part of a previous query.
26. The method of claim 1 , further comprising assigning a containerized search node of the first plurality of containerized search nodes to search a bucket of the plurality of time series buckets based at least in part on a determination that a local or shared data store associated with the containerized search node includes a copy of the bucket.
27. The method of claim 1 , wherein said executing the first query comprises:
determining that at least one bucket of the plurality of time series buckets that is assigned to a containerized search node of the first plurality of containerized search nodes is not located on the containerized search node; and
based at least in part on determining of retrieving the at least one bucket from the remote shared storage system.
28. The method of claim 1 , wherein at least one containerized search node of the group of containerized search nodes is included in the first plurality of containerized search nodes and in the second plurality of containerized search nodes.
29. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a first query identifying a first set of data to be processed and a manner of processing the first set of data;
identify a plurality of time series buckets that stores the first set of data based at least in part on a respective identifier for each time series bucket of the plurality of time series buckets, wherein the plurality of time series buckets is located in a remote shared storage system, and wherein each time series bucket of the plurality of time series buckets stores raw machine data associated with a timestamp;
identify a first plurality of containerized search nodes from a group of containerized search nodes to execute the first query, wherein the group of containerized search nodes are implemented as distinct containers in a shared computing resource environment and wherein each time series bucket of the plurality of time series buckets is accessible by any one of the group of containerized search nodes;
execute the first query using the first plurality of containerized search nodes;
store results of the first query in a data store for combination with additional query results, wherein the data store is accessible by any one of the group of containerized search nodes;
receive a second query identifying a second set of data to be processed and a manner of processing the second set of data;
identify a second plurality of containerized search nodes from the croup of containerized search nodes to execute the second query; and
execute the second query using the second plurality of containerized search nodes, wherein the executing the second query comprises combining at least a portion of results of the second query with at least a portion of the results of the first query.
30. A computing system, comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a first query identifying a first set of data to be processed and a manner of processing the first set of data;
identify a plurality of time series buckets that stores the first set of data based at least in part on a respective identifier for each time series bucket of the plurality of time series buckets, wherein the plurality of time series buckets is located in a remote shared storage system, and wherein each time series bucket of the plurality of time series buckets stores raw machine data associated with a timestamp;
identify a first plurality of containerized search nodes from a group of containerized search nodes to execute the first query, wherein the group of containerized search nodes are implemented as distinct containers in a shared computing resource environment and wherein each time series bucket of the plurality of time series buckets is accessible by any one of the group of containerized search nodes;
execute the first query using the first plurality of containerized search nodes;
store results of the first query in a data store for combination with additional query results, wherein the data store is accessible by any one of the group of containerized search nodes;
receive a second query identifying a second set of data to be processed and a manner of processing the second set of data;
identify a second plurality of containerized search nodes from the group of containerized search nodes to execute the second query; and
execute the second query using the second plurality of containerized search nodes, wherein the executing the second query comprises combining at least a portion of results of the second query with at least a portion of the results of the first query.
31. The system of claim 30 , wherein the remote shared storage system is cloud storage.
32. The system of claim 30 , wherein the remote shared storage system is S3 storage.
33. The system of claim 30 , wherein multiple copies of the plurality of time series buckets are stored in the remote shared storage system.
34. The system of claim 30 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog.
35. The system of claim 30 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog that further includes location information for each time series bucket of the plurality of time series buckets.
36. The system of claim 30 , wherein the respective identifier for each time series bucket of the plurality of time series buckets are stored in a data store catalog that further includes a time range associated with, a tenant associated with, a partition associated with, and location information for, each time series bucket of the plurality of time series buckets.
37. The system of claim 30 , wherein the one or more processing devices are further configured to assign a containerized search node of the first plurality of containerized search nodes to search a bucket of the plurality of time series buckets based at least in part on a determination that the containerized search node was assigned to search the bucket as part of a previous query.