External dataset capability compensation
Systems and methods are disclosed for processing queries against an external data source utilizing dynamically allocated partitions operating on one or more worker nodes. The external data source can include data that has not been processed by the system. To query the external data source, a query coordinator can generate a subquery for the external data source based on determined functionality of the data source. The subquery can identify data in the external data source for processing and a manner for processing the data. In addition, the query coordinator can dynamically allocate partitions operating on worker nodes to retrieve and intake results of the subquery. In some cases, number of partitions allocated can be based on a number of partitions supported by the external data source.
1. A method, comprising:
receiving, at a data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data, wherein the data intake and query system comprises a first plurality of processors;
defining a query processing scheme for obtaining and processing the set of data, wherein defining the query processing scheme comprises:
identifying, by the data intake and query system, an external data source that stores at least a portion of the set of data, wherein the external data source is distinct from the data intake and query system,
determining, by the data intake and query system, data handling capabilities of the external data source,
generating, by the data intake and query system, a subquery for the external data source based on the data handling capabilities of the external data source, the subquery identifying the at least a portion of the set of data stored in the external data source and a manner of processing the at least a portion of the set of data,
determining, by the data intake and query system, prior to execution of the query, that the external data source is configured to support a number of concurrent data connections for parallel data transport of results of the subquery from the external data source to a second plurality of processors in communication with a component of the data intake and query system, and
dynamically allocating, by the data intake and query system, a subset of the second plurality of processors for the subquery based at least in part on the number of concurrent data connections that the external data source is configured to support;
generating instructions, by the data intake and query system, for the subset of the second plurality of processors to:
provide the subquery to the external data source,
receive the results of the subquery in parallel from the external data source based at least in part on the number of concurrent data connections that the external data source is configured to support,
process the results of the subquery to generate processed results, and
provide the processed results to the component of the data intake and query system; and
executing the query based at least in part on the query processing scheme, wherein executing the query comprises communicating the instructions to the subset of the second plurality of processors.
2. The method of claim 1 , wherein defining the query processing scheme further comprises monitoring the external data source for activity and accessibility.
3. The method of claim 1 , wherein the number of concurrent data connections that the external data source is configured to support indicates that the external data source is configured to support a single parallel read, wherein the subquery comprises a plurality of subqueries, and wherein each processor of the second plurality of processors is configured to communicate a subquery of the plurality of subqueries to the external data source using the single parallel read.
4. The method of claim 1 , wherein the second plurality of processors comprise a processor for each concurrent data connection that the external data source is configured to support.
5. The method of claim 1 , wherein the data handling capabilities of the external data source comprises an identification of query commands supported by the external data source.
6. The method of claim 1 , wherein defining the data handling capabilities of the external data source comprises an identification of a location of processing nodes of the external data source.
7. The method of claim 1 , wherein generating the subquery comprises translating at least a portion of the query into commands understood by the external data source.
8. The method of claim 1 , wherein the subquery is generated based on an identification of commands supported by the external data source.
9. The method of claim 1 , wherein executing the query comprises monitoring the external data source.
10. The method of claim 1 , wherein executing the query comprises monitoring the external data source and allocating an additional processor to the subset of the second plurality of processors based on a determination that an additional concurrent data connection for parallel data transport is available on the external data source.
11. The method of claim 1 , wherein executing the query comprises monitoring the external data source and deallocating a processor of the subset of the second plurality of processors based on a determination that a concurrent data connection for parallel data transport of the external data source is not available.
12. The method of claim 1 , wherein executing the query further comprises monitoring the second plurality of processors.
13. The method of claim 1 , wherein defining the query processing scheme further comprises dynamically allocating the first plurality of processors to receive and process data from the subset of the second plurality of processors.
14. The method of claim 1 , wherein defining the query processing scheme further comprises allocating the first plurality of processors to receive data from the subset of the second plurality of processors based on the number of concurrent data connections that the external data source is configured to support.
15. The method of claim 1 , wherein defining the query processing scheme further comprises dynamically allocating the first a plurality of processors to receive data from the subset of the second plurality of processors and generating additional instructions for execution by the first plurality of processors.
16. The method of claim 1 , wherein defining the query processing scheme further comprises dynamically allocating the first plurality of processors to receive data from the subset of the second plurality of processors and generating additional instructions for execution by the first plurality of processors, and wherein executing the query further comprises communicating the additional instructions to the first plurality of processors.
17. The method of claim 1 , wherein defining the query processing scheme further comprises dynamically allocating the first plurality of processors to receive data from the subset of the second plurality of processors, and wherein executing the query further comprises monitoring the second plurality of processors and the first plurality of processors during execution of the query.
18. The method of claim 1 , wherein defining the query processing scheme further comprises dynamically allocating the first plurality of processors to receive and process data from the subset of the second plurality of processors and dynamically allocating a third plurality of processors to collect data from the first plurality of processors.
19. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions for the second plurality of processors.
20. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions for the second plurality of processors, and wherein executing the query further comprises communicating the directed acyclic graph instructions to the second plurality of processors.
21. A computing system, comprising:
one or more processing devices configured to:
receive a query identifying a set of data to be processed and a manner of processing the set of data;
define a query processing scheme for obtaining and processing the set of data, wherein to define the query processing scheme the one or more processing devices are configured to:
identify an external data source that stores at least a portion of the set of data, wherein the external data source is distinct from the computing system,
determine data handling capabilities of the external data source,
generate a subquery for the external data source based on the data handling capabilities of the external data source, the subquery identifying the at least a portion of the set of data stored in the external data source and a manner of processing the at least a portion of the set of data,
determine, prior to execution of the query, that the external data source is configured to support a number of concurrent data connections for parallel data transport of results of the subquery from the external data source to a plurality of processing devices in communication with a component of the computing system, and
dynamically allocate a subset of the plurality of processing devices for the subquery based at least in part on the number of concurrent data connections that the external data source is configured to support;
generate instructions for the subset of the plurality of processing devices to:
provide the subquery to the external data source,
receive the results of the subquery in parallel from the external data source based at least in part on the number of concurrent data connections that the external data source is configured to support,
process the results of the subquery to generate processed results, and
provide the processed results to the component of the computing system; and
execute the query based at least in part on the query processing scheme, wherein to execute the query, the one or more processing devices are configured to communicate the instructions to the subset of the plurality of processing devices.
22. The system of claim 21 , wherein the number of concurrent data connections for parallel data transport that the external data source is configured to support indicates that the external data source is configured to support a single concurrent data connection for parallel data transport, wherein the subquery comprises a plurality of subqueries, and wherein each processing device of the subset of the plurality of processing devices is configured to communicate a subquery of the plurality of subqueries to the external data source using the single concurrent data connection.
23. The system of claim 21 , wherein to execute the query, the one or more processing devices are further configured to monitor the external data source and allocate an additional processing device to the subset of the plurality of processing devices based on a determination that an additional concurrent data connection for parallel data transport is available on the external data source.
24. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive a query identifying a set of data to be processed and a manner of processing the set of data;
define a query processing scheme for obtaining and processing the set of data, wherein to define the query processing scheme the computer-executable instructions cause the computing system to:
identify an external data source that stores at least a portion of the set of data, wherein the external data source is distinct from the computing system,
determine data handling capabilities of the external data source,
generate a subquery for the external data source based on the data handling capabilities of the external data source, the subquery identifying the at least a portion of the set of data stored in the external data source and a manner of processing the at least a portion of the set of data,
determine, prior to execution of the query, that the external data source is configured to support a number of concurrent data connections for parallel data transport of results of the subquery from the external data source to a plurality of processors in communication with a component of the computing system, and
dynamically allocate a subset of the plurality of processors for the subquery based at least in part on the number of concurrent data connections that the external data source is configured to support;
generate instructions for the subset of the plurality of processors to:
provide the subquery to the external data source,
receive the results of the subquery in parallel from the external data source based at least in part on the number of concurrent data connections that the external data source is configured to support,
process the results of the subquery to generate processed results, and
provide the processed results to the component of the computing system; and
execute the query based on the query processing scheme, wherein to execute the query, the computer-executable instructions cause the computing system to communicate the instructions to the subset of the plurality of processors.
25. The method of claim 1 , wherein the data handling capabilities of the external data source comprises: an amount of data stored in the external data source, a type of data stored in an external data source, query commands supported by the external data source, a query translator supported by the external data source, or a file system type and hierarchy of the external data source.
26. The system of claim 21 , wherein the data handling capabilities of the external data source comprises: an amount of data stored in the external data source, a type of data stored in an external data source, query commands supported by the external data source, a query translator supported by the external data source, or a file system type and hierarchy of the external data source.
27. The non-transitory computer readable media of claim 24 , wherein the data handling capabilities of the external data source comprises: an amount of data stored in the external data source, a type of data stored in an external data source, query commands supported by the external data source, a query translator supported by the external data source, or a file system type and hierarchy of the external data source.