Reassigning data groups based on activation of a processing node
A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a first group of processing nodes and cause a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify a second group of processing nodes that includes the first group of processing nodes and a second processing node. The data intake and query system can transmit commands to cancel one or more data group downloads at the first processing node and can reassign the corresponding data groups for download to the second processing node based on a second node map.
1. A method, comprising:
identifying a first group of processing nodes, wherein the first group of processing nodes is associated with a plurality of data groups, wherein the plurality of data groups comprises a first data group and a second data group;
transmitting a first command to a first processing node of the first group of processing nodes to download the first data group and the second data group based at least in part on a first node map, wherein the first processing node queues, in a data queue of the first processing node, the first data group and the second data group for download based at least in part on the first command;
identifying a second group of processing nodes, the second group of processing nodes comprising the first group of processing nodes and a second processing node, wherein the second processing node is assigned the first data group and the second data group based at least in part on a second node map;
determining that the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node;
transmitting a second command to the first processing node based at least in part on determining that the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node, wherein the second command instructs the first processing node to cancel, in the data queue of the first processing node, download of at least a portion of the first data group; and
causing the second processing node to download the first data group and the second data group based at least in part on the second node map and based at least in part on transmitting the second command to the first processing node.
2. The method of claim 1 , further comprising receiving a response from the first processing node, wherein the response indicates receipt of the second command by the first processing node, wherein download of the second data group is removed from the data queue of the first processing node based at least in part on the download of the second data group by the first processing node.
3. The method of claim 1 , further comprising receiving a response from the first processing node, wherein the response indicates receipt of the second command by the first processing node, and wherein causing the second processing node to download the first data group and the second data group is further based at least in part on receiving the response.
4. The method of claim 1 , further comprising:
determining a response from the first processing node is not received within a time period;
removing the first processing node from the second group of processing nodes based at least in part on determining the response is not received within the time period; and
assigning the plurality of data groups to the second group of processing nodes.
5. The method of claim 1 , further comprising transmitting a third command to the first processing node based at least in part on determining that at least a portion of the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node, wherein the third command instructs the first processing node to remove the second data group downloaded by the first processing node.
6. The method of claim 1 , further comprising receiving one or more heartbeats from the first processing node, wherein the one or more heartbeats indicate a download status of the download of at least one of the first data group or the second data group.
7. The method of claim 1 , wherein the second data group downloaded by the first processing node is maintained by the first processing node, and wherein the second data group downloaded by the second processing node is maintained by the second processing node.
8. The method of claim 1 , wherein causing the second processing node to download the first data group and the second data group comprises transmitting a third command to the second processing node based at least in part on assigning the second processing node to download the first data group and the second data group, wherein the third command instructs the second processing node to download the first data group and the second data group.
9. The method of claim 1 , wherein causing the second processing node to download the first data group and the second data group comprises transmitting a third command to the second processing node based at least in part on assigning the second processing node to download the first data group and the second data group, wherein the third command instructs the second processing node to download the first data group and the second data group, the method further comprising:
transmitting a fourth command to a third processing node of the second group of processing nodes, wherein the fourth command instructs the third processing node to download a third data group of the plurality of data groups.
10. The method of claim 1 , further comprising:
transmitting a third command to a third processing node of the second group of processing nodes based at least in part on identifying the second group of processing nodes, wherein the third command instructs the third processing node to cancel, in a data queue of the third processing node, download of a third data group of the plurality of data groups; and
causing a fourth processing node of the second group of processing nodes to download the third data group based at least in part on transmitting the third command to the third processing node.
11. The method of claim 1 , wherein the second command preempts the first command.
12. The method of claim 1 , wherein one or more processing nodes of the first group of processing nodes are associated with a respective processing node status.
13. The method of claim 1 , further comprising monitoring a processing node status of one or more processing nodes of the first group of processing nodes.
14. The method of claim 1 , wherein one or more processing nodes of the first group of processing nodes are associated with a respective processing node status, wherein the respective processing node status comprises a download status.
15. The method of claim 1 , wherein the plurality of data groups comprises a plurality of partitions, each partition of the plurality of partitions comprising one or more buckets of data.
16. The method of claim 1 , further comprising causing the second processing node to execute one or more searches on at least one of the first data group or the second data group based at least in part on causing the second processing node to download the first data group and the second data group.
17. The method of claim 1 , further comprising causing the first processing node to download a third data group of the plurality of data groups based at least in part on identifying the second group of processing nodes, wherein the first processing node is instructed to not download the first data group and the second data group based at least in part on identifying the second group of processing nodes.
18. The method of claim 1 , further comprising identifying activation of the second processing node, wherein identifying the second group of processing nodes is based at least in part on identifying activation of the second processing node.
19. A system comprising:
a data store; and
one or more processors configured to:
identify a first group of processing nodes, wherein the first group of processing nodes is associated with a plurality of data groups, wherein the plurality of data groups comprises a first data group and a second data group;
transmit a first command to a first processing node of the first group of processing nodes to download the first data group and the second data group based at least in part on a first node map, wherein the first processing node queues, in a data queue of the first processing node, the first data group and the second data group for download based at least in part on the first command;
identify a second group of processing nodes, the second group of processing nodes comprising the first group of processing nodes and a second processing node, wherein the second processing node is assigned the first data group and the second data group based at least in part on a second node map;
determine that the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node;
transmit a second command to the first processing node based at least in part on determining that the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node, wherein the second command instructs the first processing node to cancel, in the data queue of the first processing node, download of at least a portion of the first data group; and
cause the second processing node to download the first data group and the second data group based at least in part on the second node map and based at least in part on transmitting the second command to the first processing node.
20. Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to:
identify a first group of processing nodes, wherein the first group of processing nodes is associated with a plurality of data groups, wherein the plurality of data groups comprises a first data group and a second data group;
transmit a first command to a first processing node of the first group of processing nodes to download the first data group and the second data group based at least in part on a first node map, wherein the first processing node queues, in a data queue of the first processing node, the first data group and the second data group for download based at least in part on the first command;
identify a second group of processing nodes, the second group of processing nodes comprising the first group of processing nodes and a second processing node, wherein the second processing node is assigned the first data group and the second data group based at least in part on a second node map;
determine that the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node;
transmit a second command to the first processing node based at least in part on determining that the first data group is queued in the data queue of the first processing node and that the second data group is downloaded by the first processing node, wherein the second command instructs the first processing node to cancel, in the data queue of the first processing node, download of at least a portion of the first data group; and
cause the second processing node to download the first data group and the second data group based at least in part on the second node map and based at least in part on transmitting the second command to the first processing node.