Report acceleration using intermediate results in a distributed indexer system for searching events
A method and system for managing searches of a data set that is partitioned based on a plurality of events. A structure of a search query may be analyzed to determine if logical computational actions performed on the data set is reducible. Data in each partition is analyzed to determine if at least a portion of the data in the partition is reducible. In response to a subsequent or reoccurring search request, intermediate summaries of reducible data and reducible search computations may be aggregated for each partition. Next, a search result may be generated based on at least one of the aggregated intermediate summaries, the aggregated reducible search computations, and a query of adhoc non-reducible data arranged in at least one of the plurality of partitions for the data set.
1. A computer-implemented method, comprising:
receiving data on a computing device;
parsing the data to create a set of events, wherein each event in the set of events includes a portion of the data;
associating a time with each event in the set of events;
storing the set of events in a partition to which an indexer in a set of distributed indexers has access;
calculating, by the indexer, an intermediate result on data extracted from the set of events in the partition to improve response time for a recurring query whose recurrences are not all directed to identical time periods, the intermediate result including information used to satisfy two or more recurrences of the recurring query, wherein the times associated with the set of events in the partition are included in both a first time period and a second time period that only partially overlaps with the first time period;
producing a first report covering the first time period by combining a partial result produced for the first time period by the indexer that has access to the partition with a partial result produced for the first time period by at least one other indexer, wherein the partial result for the first time period that is produced by the indexer with access to the partition is determined using the intermediate result; and
producing a second report covering the second time period by combining a partial result produced for the second time period by the indexer with access to the partition with a partial result produced for the second time period by at least one other indexer, wherein the partial result for the second time period that is produced by the indexer with access to the partition is determined using the intermediate result and without further using the set of events used to calculate the intermediate result.
2. The method of claim 1 , wherein the partition is selected for storing the set of events based on the times associated with the events in the set of events.
3. The method of claim 1 , wherein the partial result produced for the first time period by the indexer that has access to the partition and the partial result produced for the first time period by the at least one other indexer are generated concurrently.
4. The method of claim 1 , wherein an end of the second time period is later than an end of the first time period, and wherein the second report is produced after the first report is produced.
5. The method of claim 1 , wherein the first report and the second report are generated based on a same query.
6. The method of claim 1 , wherein the intermediate result is calculated based on determining, by the indexer, that a calculation needed to produce the first report and the second report is reusable.
7. The method of claim 1 , wherein the intermediate result is calculated based on determining, by the indexer, that data in the set of events in the partition is reusable for purposes of generating both the first report and the second report.
8. The method of claim 1 , wherein the intermediate result is calculated based on determining that data in the set of events in the partition can be summarized for purposes of generating both the first report and the second report.
9. The method of claim 1 , wherein storing the set of events in the partition to which the indexer in the set of distributed indexers has access includes selecting the partition based on the time associated with the events in the set of events.
10. The method of claim 1 , wherein the partial result for the second time period that is produced by the indexer with access to the partition is determined using both the intermediate result and using data extracted from events in the partition that were not used to calculate the intermediate result.
11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computer devices, cause performance of:
receiving data on a computing device;
parsing the data to create a set of events, wherein each event in the set of events includes a portion of the data;
associating a time with each event in the set of events;
storing the set of events in a partition to which an indexer in a set of distributed indexers has access;
calculating, by the indexer, an intermediate result on data extracted from the set of events in the partition to improve response time for a recurring query whose recurrences are not all directed to identical time periods, the intermediate result including information used to satisfy two or more recurrences of the recurring query, wherein the times associated with the set of events in the partition are included in both a first time period and a second time period that only partially overlaps with the first time period;
producing a first report covering the first time period by combining a partial result produced for the first time period by the indexer that has access to the partition with a partial result produced for the first time period by at least one other indexer, wherein the partial result for the first time period that is produced by the indexer with access to the partition is determined using the intermediate result; and
producing a second report covering the second time period by combining a partial result produced for the second time period by the indexer with access to the partition with a partial result produced for the second time period by at least one other indexer, wherein the partial result for the second time period that is produced by the indexer with access to the partition is determined using the intermediate result and without further using the set of events used to calculate the intermediate result.
12. The one or more non-transitory computer-readable media of claim 11 , wherein an end of the second time period is later than an end of the first time period, and wherein the second report is produced after the first report is produced.
13. The one or more non-transitory computer-readable media of claim 11 , wherein the first report and the second report are generated based on a same query.
14. The one or more non-transitory computer-readable media of claim 11 , wherein the intermediate result is calculated based on determining, by the indexer, that a calculation needed to produce the first report and the second report is reusable.
15. The one or more non-transitory computer-readable media of claim 11 , wherein the intermediate result is calculated based on determining, by the indexer, that data in the set of events in the partition is reusable for purposes of generating both the first report and the second report.
16. An apparatus, comprising:
an indexer;
a data receiver at a computing device, implemented at least partially in hardware, that receives data;
a data parser at the computing device, implemented at least partially in hardware, that parses the data to create a set of events, wherein each event in the set of events includes a portion of the data;
a time association subsystem at the computing device, implemented at least partially in hardware, that associates a time with each event in the set of events;
a storage subsystem at the computing device, implemented at least partially in hardware, that stores the set of events in a partition to which an indexer in a set of distributed indexers has access;
an intermediate result calculation subsystem at the indexer, implemented at least partially in hardware, that calculates an intermediate result on data extracted from the set of events in the partition to improve response time for a recurring query whose recurrences are not all directed to identical time periods, the intermediate result including information used to satisfy two or more recurrences of the recurring query, wherein the times associated with the set of events in the partition are included in both a first time period and a second time period that only partially overlaps with the first time period;
a report production subsystem, implemented at least partially in hardware, that produces a first report covering the first time period by combining a partial result produced for the first time period by the indexer that has access to the partition with a partial result produced for the first time period by at least one other indexer, wherein the partial result for the first time period that is produced by the indexer with access to the partition is determined using the intermediate result; and
wherein the report production subsystem produces a second report covering the second time period by combining a partial result produced for the second time period by the indexer with access to the partition with a partial result produced for the second time period by at least one other indexer, wherein the partial result for the second time period that is produced by the indexer with access to the partition is determined using the intermediate result and without further using the set of events used to calculate the intermediate result.
17. The apparatus of claim 16 , wherein an end of the second time period is later than an end of the first time period, and wherein the second report is produced after the first report is produced.
18. The apparatus of claim 16 , wherein the first report and the second report are generated based on a same query.
19. The apparatus of claim 16 , wherein the intermediate result is calculated based on determining, by the indexer, that a calculation needed to produce the first report and the second report is reusable.
20. The apparatus of claim 16 , wherein the intermediate result is calculated based on determining, by the indexer, that data in the set of events in the partition is reusable for purposes of generating both the first report and the second report.