Parallel branch operation using intermediary nodes
The disclosed implementations include a method performed by a data intake and query system. The method includes receiving a search query at a search head, the search query including a branching operation between sets of data, generating a first subquery and a second subquery corresponding to the sets of data for execution by a search node, generating instructions for an intermediary node to combine partial results of the first subquery and the second subquery and instructions to concurrently communicate the subqueries to a search node, and executing the query by providing the instructions for the intermediary node to the intermediary node and the subqueries to the search node, the intermediary node receiving sets of partial search results for the subqueries, performing at least a portion of the branching operation on the partial results, and communicating the combined results to another intermediary node or the search head.
1. A method, comprising:
receiving a query at a search head of a data intake and query system;
parsing the query;
based on parsing the query:
determining that the query includes a join command to join a first set of data and a second set of data, and
in response to determining that the query includes the join command, identifying a first portion of the query as a first subquery and a second portion of the query as a second subquery, wherein the first subquery corresponds to the first set of data and the second subquery corresponds to the second set of data;
identifying at least a first search node and a second search node of a plurality of search nodes instantiated within the data intake and query system based on determining that the query includes the join command;
generating instructions for a first intermediary node of the data intake and query system to combine partial results of the first subquery and the second subquery as at least part of a join operation;
generating instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node;
executing the query, wherein executing the query includes communicating the instructions for the first intermediary node to the first intermediary node and concurrently transmitting each of the first subquery and the second subquery to each of the first search node and the second search node,
wherein each of the first search node and the second search node concurrently executes each of the first subquery and the second subquery,
wherein each of the first search node and the second search node identifies respective first partial results based on execution of the first subquery and respective second partial results based on execution of the second subquery,
wherein each of the first search node and the second search node provides the respective first partial results and the respective second partial results to the first intermediary node,
wherein the first intermediary node concurrently receives the respective first partial results and the respective second partial results from each of the first search node and the second search node, and combines the respective first partial results and the respective second partial results from each of the first search node and the second search node as at least part of the join operation, and communicates results of the join operation to at least one of a second intermediary node or the search head; and
receiving, at the search head, results of the query from at least one of the first intermediary node, the second intermediary node, or another intermediary node.
2. The method of claim 1 , wherein the first intermediary node is a search node.
3. The method of claim 1 , wherein the second intermediary node receives the results of the join operation from the first intermediary node and combines the results of the join operation as at least part of the join operation.
4. The method of claim 1 , wherein the first subquery and the second subquery include instructions for each of the first search node and the second search node to provide the respective first partial results and the respective second partial results to the first intermediary node.
5. The method of claim 1 , wherein the first intermediary node obtains the instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node, wherein the first intermediary node concurrently transmits each of the first subquery and the second subquery to each of the first search node and the second search node.
6. The method of claim 1 , wherein the search head obtains the instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node, wherein the search head concurrently transmits each of the first subquery and the second subquery to each of the first search node and the second search node.
7. The method of claim 1 , wherein the search head communicates the instructions for the first intermediary node, the first subquery, and the second subquery to each of the first search node and the second search node, wherein the first search node and the second search node:
select the first intermediary node to combine the respective first partial results and the respective second partial results from each of the first search node and the second search node; and
communicate the instructions for the first intermediary node, the respective first partial results from each of the first search node and the second search node, and the respective second partial results from each of the first search node and the second search node to the first intermediary node.
8. The method of claim 1 , further comprising selecting the first intermediary node based at least in part on one or more of a capacity of the first intermediary node or an availability of the first intermediary node.
9. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a query at a search head of a data intake and query system;
parse the query;
based on parsing the query:
determine that the query includes a join command to join a first set of data and a second set of data, and
in response to determining that the query includes the join command,
identify a first portion of the query as a first subquery and a second portion of the query as a second subquery, wherein the first subquery corresponds to the first set of data and the second subquery corresponds to the second set of data;
identify at least a first search node and a second search node of a plurality of search nodes instantiated within the data intake and query system based on determining that the query includes the join command;
generate instructions for first intermediary node of the data intake and query system to combine partial results of the first subquery and the second subquery as at least part of a join operation;
generate instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node;
execute the query, wherein executing the query includes communicating the instructions for the first intermediary node to the first intermediary node and concurrently transmitting each of the first subquery and the second subquery to each of the first search node and the second search node,
wherein each of the first search node and the second search node concurrently executes each of the first subquery and the second subquery,
wherein each of the first search node and the second search node identifies respective first partial results based on execution of the first subquery and respective second partial results based on execution of the second subquery,
wherein each of the first search node and the second search node provides the respective first partial results and the respective second partial results to the first intermediary node,
wherein the first intermediary node concurrently receives the respective first partial results and the respective second partial results from each of the first search node and the second search node, and combines the respective first partial results and the respective second partial results from each of the first search node and the second search node as at least part of the join operation, and communicates results of the join operation to at least one of a second intermediary node or the search head; and
receive, at the search head, results of the query from at least one of the first intermediary node, the second intermediary node, or another intermediary node.
10. The non-transitory computer readable media of claim 9 , wherein the first intermediary node is a search node.
11. The non-transitory computer readable media of claim 9 , wherein the second intermediary node receives the results of the join operation from the first intermediary node and combines the results of the join operation as at least part of the join operation.
12. The non-transitory computer readable media of claim 9 , wherein the first subquery and the second subquery include instructions for each of the first search node and the second search node to provide the respective first partial results and the respective second partial results to the first intermediary node.
13. The non-transitory computer readable media of claim 9 , wherein the first intermediary node obtains the instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node, wherein the first intermediary node concurrently transmits each of the first subquery and the second subquery to each of the first search node and the second search node.
14. The non-transitory computer readable media of claim 9 , wherein the search head obtains the instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node, wherein the search head concurrently transmits each of the first subquery and the second subquery to each of the first search node and the second search node.
15. The non-transitory computer readable media of claim 9 , wherein the search head communicates the instructions for the first intermediary node, the first subquery, and the second subquery to each of the first search node and the second search node, wherein the first search node and the second search node:
select the first intermediary node to combine the respective first partial results and the respective second partial results from each of the first search node and the second search node; and
communicate the instructions for the first intermediary node, the respective first partial results from each of the first search node and the second search node, and the respective second partial results from each of the first search node and the second search node to the first intermediary node.
16. The non-transitory computer readable media of claim 9 , wherein the computer-executable instructions, when executed by the computing system of the data intake and query system, further cause the computing system to select the first intermediary node based at least in part on one or more of a capacity of the first intermediary node or an availability of the first intermediary node.
17. A computing system, comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a query at a search head of a data intake and query system;
parse the query;
based on parsing the query:
determine that the query includes a join command to join a first set of data and a second set of data, and
in response to determining that the query includes the join command, identify a first portion of the query as a first subquery and a second portion of the query as a second subquery, wherein the first subquery corresponds to the first set of data and the second subquery corresponds to the second set of data;
identify at least a first search node and a second search node of a plurality of search nodes instantiated within the data intake and query system based on determining that the query includes the join command;
generate instructions for a first intermediary node of the data intake and query system to combine partial results of the first subquery and the second subquery as at least part of a join operation;
generate instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node;
execute the query, wherein executing the query includes communicating the instructions for the first intermediary node to the first intermediary node and concurrently transmitting each of the first subquery and the second subquery to each of the first search node and the second search node,
wherein each of the first search node and the second search node concurrently executes each of the first subquery and the second subquery,
wherein each of the first search node and the second search node identifies respective first partial results based on execution of the first subquery and respective second partial results based on execution of the second subquery,
wherein each of the first search node and the second search node provides the respective first partial results and the respective second partial results to the first intermediary node,
wherein the first intermediary node concurrently receives the respective first partial results and the respective second partial results from each of the first search node and the second search node, and combines the respective first partial results and the respective second partial results from each of the first search node and the second search node as at least part of the join operation, and communicates results of the join operation to at least one of a second intermediary node or the search head; and
receive, at the search head, results of the query from at least one of the first intermediary node, the second intermediary node, or another intermediary node.
18. The system of claim 17 , wherein the search head obtains the instructions to concurrently transmit each of the first subquery and the second subquery to each of the first search node and the second search node, wherein the search head concurrently transmits each of the first subquery and the second subquery to each of the first search node and the second search node.
19. The system of claim 17 , wherein the search head communicates the instructions for the first intermediary node, the first subquery, and the second subquery to each of the first search node and the second search node, wherein the first search node and the second search node:
select the first intermediary node to combine the respective first partial results and the respective second partial results from each of the first search node and the second search node; and
communicate the instructions for the first intermediary node, the respective first partial results from each of the first search node and the second search node, and the respective second partial results from each of the first search node and the second search node to the first intermediary node.
20. The system of claim 17 , wherein one or more processors are further configured to select the first intermediary node based at least in part on one or more of a capacity of the first intermediary node or an availability of the first intermediary node.