Reassigning processing tasks to an external storage system
Systems and methods are described for reducing execution time of a query that references external data systems. The system can determine an external data system is capable of processing one or more map or reduce phases of a map-reduce operation. When it is determined that the external data system can process a map or reduce phase, associated operations may be reassigned from the system to the external data system reducing the processing resources used by the system to response to the query and, in some cases, speeding up performance of the query.
1. A method, comprising:
receiving, at a data intake and query system comprising one or more hardware processors, a query;
determining, by accessing a configuration file, that the query references an external search for a remote data store that stores data referenced by the external search, wherein the external search comprises a subquery of the query that is identified from the configuration file based at least in part on a reference within the query;
determining that the remote data store includes processing capabilities;
determining a type of processing capabilities of the remote data store;
identifying additional processing to be performed on results of the external search based on the query;
determining an amount of time to perform the additional processing at the remote data store;
determining that the amount of time to perform the additional processing at the remote data store does not exceed a threshold time, wherein the threshold time is based at least in part on an amount of time to perform the additional processing at the data intake and query system, and wherein the threshold time is based at least in part on an amount of computing resources available at the data intake and query system to execute the additional processing; and
in response to determining that the amount of time to perform the additional processing at the remote data store does not exceed the threshold time,
generating additional processing instructions based on the identified additional processing to be performed on the results of the external search and the determined type of processing capabilities of the remote data store; and
communicating the external search in combination with the additional processing instructions to the remote data store to perform preprocessing of search results of the external search to obtain a modified result of the external search to be received from the remote data store in place of the search results, wherein the amount of time to perform the additional processing at the remote data store further comprises an amount of time to transmit the modified result of the external search to the data intake and query system, and wherein the modified result of the external search comprises a reduced data set and wherein the amount of time to transmit the modified result of the external search accounts for a reduced transmission time to transmit the reduced data set.
2. The method of claim 1 , wherein determining that the remote data store includes processing capabilities comprises determining that the remote data store exposes at least some of the processing capabilities for access by an external system.
3. The method of claim 1 , wherein determining the type of processing capabilities of the remote data store further comprises determining whether the remote data store is capable of performing a map operation, a reduce operation, or both the map operation and the reduce operation in a map-reduce processing model.
4. The method of claim 1 , wherein determining the type of processing capabilities of the remote data store further comprises determining whether the remote data store can perform a set of operations in a map-reduce processing model without causing processing time to exceed the threshold time.
5. The method of claim 1 , wherein the configuration file stores an identity of a set of queries performable at one or more remote systems.
6. The method of claim 1 , wherein the additional processing instructions comprise instructions to perform a reduce operation when it is determined that the type of processing capabilities includes the capability of performing map operations and reduce operations.
7. The method of claim 1 , wherein the additional processing instructions comprise instructions to perform a map operation of a map-reduce processing model while omitting a reduce operation of the map-reduce processing model when it is determined that the type of processing capabilities includes the capability of performing map operations and not reduce operations.
8. The method of claim 1 , wherein, when determining that the type of processing capabilities of the remote data store includes the capability to perform reduce operations, the method further comprises:
determining that executing the query includes performing a reduce operation with respect to data stored at the remote data store; and
determining a processing time to perform the reduce operation at the remote data store.
9. The method of claim 8 , wherein the additional processing instructions comprise instructions to perform the reduce operation at the remote data store when it is determined that the processing time at the remote data store does not exceed a threshold time.
10. The method of claim 8 , wherein the additional processing instructions omits instructions to perform the reduce operation at the remote data store when it is determined that the processing time at the remote data store exceeds a threshold time.
11. The method of claim 8 , wherein determining the processing time to perform the reduce operation comprises estimating the processing time based at least in part on an estimate of the size of the data at the remote data store.
12. The method of claim 8 , wherein the reduce operation comprises a filtering operation or an aggregation operation.
13. The method of claim 1 , wherein communicating the external search in combination with the additional processing instructions comprises transmitting the external search in combination with the additional processing instructions to the remote data store to be executed by the remote data store.
14. The method of claim 1 , wherein communicating the external search in combination with the additional processing instructions comprises transmitting the external search in combination with the additional processing instructions to a worker node comprising one or more processors configured to interact with the remote data store.
15. The method of claim 1 , wherein at least a portion of the query is executed at a data intake and query system and wherein communicating the additional processing instructions to the remote data store or a worker node reduces the processing performed at the data intake and query system.
16. The method of claim 1 , wherein at least a portion of the query is executed at a data intake and query system and wherein the method further comprises modifying the at least the portion of the query executed at the data intake and query system based at least in part on the additional processing instructions.
17. The method of claim 16 , wherein modifying the at least the portion of the query executed at the data intake and query system reduces the operations performed at the data intake and query system.
18. The method of claim 1 , wherein the query references a second external search for a second remote data store.
19. The method of claim 18 , wherein the method further comprises:
determining that the second remote data store does not include processing capabilities; and
directing a worker node to perform the second external search with respect to the second remote data store without generating additional processing instructions to be performed by the second remote data store.
20. The method of claim 18 , wherein the method further comprises:
determining that generating a response to the query includes performing a reduce operation on a combination of a result of the external search and a result of the second external search;
determining that the reduce operation references data fields obtained from the external search and omitted from the second external search; and
excluding performance of the second external search as a result of determining that the reduce operation references fields obtained from the external search and omitted from the second external search.
21. The method of claim 18 , wherein the external search and the second external search are performed at least partially in parallel.
22. The method of claim 18 , further comprising generating a response to the query based at least in part on a response to the external search in combination with the additional processing instructions and a response to the second external search.
23. The method of claim 1 , further comprising modifying the external search based at least in part on the additional processing instructions to obtain a modified external search, wherein communicating the external search in combination with the additional processing instructions comprises communicating the modified external search.
24. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive, at the data intake and query system comprising the one or more processing devices, a query;
determine, by accessing a configuration file, that the query references an external search for a remote data store that stores data referenced by the external search, wherein the external search comprises a subquery of the query that is identified from the configuration file based at least in part on a reference within the query;
determine that the remote data store includes processing capabilities;
determine a type of processing capabilities of the remote data store;
identify additional processing to be performed on results of the external search based on the query;
determine an amount of time to perform the additional processing at the remote data store;
determine that the amount of time to perform the additional processing at the remote data store does not exceed a threshold time, wherein the threshold time is based at least in part on an amount of time to perform the additional processing at the data intake and query system, and wherein the threshold time is based at least in part on an amount of computing resources available at the data intake and query system to execute the additional processing; and
in response to determining that the amount of time to perform the additional processing at the remote data store does not exceed the threshold time,
generate additional processing instructions based on the identified additional processing to be performed on the results of the external search and the determined type of processing capabilities of the remote data store; and
communicate the external search in combination with the additional processing instructions to the remote data store or a worker node configured to interact with the remote data store to perform preprocessing of search results of the external search to obtain a modified result of the external search to be received from the remote data store in place of the search results, wherein the amount of time to perform the additional processing at the remote data store further comprises an amount of time to transmit the modified result of the external search to the data intake and query system, and wherein the modified result of the external search comprises a reduced data set and wherein the amount of time to transmit the modified result of the external search accounts for a reduced transmission time to transmit the reduced data set.
25. The computing system of claim 24 , wherein, when determining that the type of processing capabilities of the remote data store includes the capability to perform reduce operations, the one or more processing devices are further configured to:
determine that executing the query includes performing a reduce operation with respect to data stored at the remote data store; and
determine a processing time to perform the reduce operation at the remote data store.
26. The computing system of claim 25 , wherein the additional processing instructions comprise instructions to perform the reduce operation at the remote data store when it is determined that the processing time at the remote data store does not exceed the threshold time, and wherein the additional processing instructions omits instructions to perform the reduce operation at the remote data store when it is determined that the processing time at the remote data store exceeds the threshold time.
27. The computing system of claim 24 , wherein the query references a second external search for a second remote data store.
28. The computing system of claim 27 , wherein the one or more processing devices are further configured to:
determine that the second remote data store does not include processing capabilities; and
direct the worker node to perform the second external search with respect to the second remote data store without generating additional processing instructions to be performed by the second remote data store.
29. The computing system of claim 27 , wherein the one or more processing devices are further configured to:
determine that generating a response to the query includes performing a reduce operation on a combination of a result of the external search and a result of the second external search;
determine that the reduce operation references data fields obtained from the external search and omitted from the second external search; and
exclude performance of the second external search as a result of determining that the reduce operation references fields obtained from the external search and omitted from the second external search.
30. Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive, at the data intake and query system comprising one or more hardware processors, a query;
determine, by accessing a configuration file, that the query references an external search for a remote data store that stores data referenced by the external search, wherein the external search comprises a subquery of the query that is identified from the configuration file based at least in part on a reference within the query;
determine that the remote data store includes processing capabilities;
determine a type of processing capabilities of the remote data store;
identify additional processing to be performed on results of the external search based on the query;
determine an amount of time to perform the additional processing at the remote data store;
determine that the amount of time to perform the additional processing at the remote data store does not exceed a threshold time, wherein the threshold time is based at least in part on an amount of time to perform the additional processing at the data intake and query system, and wherein the threshold time is based at least in part on an amount of computing resources available at the data intake and query system to execute the additional processing; and
in response to determining that the amount of time to perform the additional processing at the remote data store does not exceed the threshold time,
generate additional processing instructions based on the identified additional processing to be performed on the results of the external search and the determined type of processing capabilities of the remote data store; and
communicate the external search in combination with the additional processing instructions to the remote data store or a worker node configured to interact with the remote data store to perform preprocessing of search results of the external search to obtain a modified result of the external search to be received from the remote data store in place of the search results, wherein the amount of time to perform the additional processing at the remote data store further comprises an amount of time to transmit the modified result of the external search to the data intake and query system, and wherein the modified result of the external search comprises a reduced data set and wherein the amount of time to transmit the modified result of the external search accounts for a reduced transmission time to transmit the reduced data set.