GENERATING INTERIM PROCESSING NODE MAPS
A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a first group of processing nodes and cause a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify a second group of processing nodes that includes the first group of processing nodes and a second processing node. The data intake and query system can transmit commands to cancel one or more data group downloads at the first processing node and can reassign the corresponding data groups for download to the second processing node based on a second node map.
1 . A method, comprising:
identifying a new processing node to include in a set of processing nodes that process buckets associated with a set of data groups in accordance with a processing node map that associates processing nodes with data groups of the set of data groups;
based on the new processing node to include in the set of processing nodes, generating an interim processing node map to transition a new processing node into use, the interim processing node map including associations of processing nodes, including the new processing node, with the data groups of the set of data groups, wherein the new processing node corresponds with less data groups than other processing nodes of the set of processing nodes; and
processing buckets associated with the set of data groups in accordance with the interim processing node map.
2 . The method of claim 1 , wherein the interim processing node map is incrementally assigned responsibilities to the new processing node over a time duration until processing nodes, including the new processing node, include a similar amount of responsibilities.
3 . The method of claim 1 , wherein the interim processing node map is generated using a tentative node map according to a processing node map generation policy and reassigning data groups for the new node to a different processing node to generate the interim processing node map.
4 . The method of claim 1 , wherein the reassigning of data groups is based on assignments indicated in a previous processing node map.
5 . The method of claim 3 , wherein the processing node map generation policy is configured to achieve load balancing.
6 . The method of claim 3 , wherein the processing node map generation policy is configured to achieve an approximately equal distribution of groups of data.
7 . The method of claim 3 , wherein the processing node map generation policy indicates that the data groups are to be assigned to processing nodes according to a hashing algorithm.
8 . The method of claim 1 further comprising transitioning the interim processing node map to a final processing map that is generated in accordance with a map transition policy.
9 . The method of claim 8 , wherein the map transition policy indicates a transition from the interim processing node map to the final processing map based on a threshold amount of time.
10 . The method of claim 8 , wherein the map transition policy indicates a transition from the interim processing node map to the final processing map based on a quantity of cache misses.
11 . The method of claim 8 , wherein the map transition policy indicates a transition from the interim processing node map to the final processing map based on an amount of data downloaded.
12 . The method of claim 8 , wherein the interim processing node map transitions to a new interim processing node map that includes at least one additional data group for the new processing node.
13 . A system comprising:
a data store; and
one or more processors configured to:
identify a new processing node to include in a set of processing nodes that process buckets associated with a set of data groups in accordance with a processing node map that associates processing nodes with data groups of the set of data groups;
based on the new processing node to include in the set of processing nodes, generate an interim processing node map to transition a new processing node into use, the interim processing node map including associations of processing nodes, including the new processing node, with the data groups of the set of data groups, wherein the new processing node corresponds with less data groups than other processing nodes of the set of processing nodes; and
process buckets associated with the set of data groups in accordance with the interim processing node map.
14 . The system of claim 13 , wherein the interim processing node map is incrementally assigned responsibilities to the new processing node over a time duration until processing nodes, including the new processing node, include a similar amount of responsibilities.
15 . The system of claim 13 , wherein the interim processing node map is generated using a tentative node map according to a processing node map generation policy and reassigning data groups for the new node to a different processing node to generate the interim processing node map.
16 . The system of claim 13 , wherein the reassigning of data groups is based on assignments indicated in a previous processing node map.
17 . The non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to:
identify a new processing node to include in a set of processing nodes that process buckets associated with a set of data groups in accordance with a processing node map that associates processing nodes with data groups of the set of data groups;
based on the new processing node to include in the set of processing nodes, generate an interim processing node map to transition a new processing node into use, the interim processing node map including associations of processing nodes, including the new processing node, with the data groups of the set of data groups, wherein the new processing node corresponds with less data groups than other processing nodes of the set of processing nodes; and
process buckets associated with the set of data groups in accordance with the interim processing node map.
18 . The non-transitory computer-readable media of claim 17 further comprising transitioning the interim processing node map to a final processing map that is generated in accordance with a map transition policy.
19 . The non-transitory computer-readable media of claim 18 , wherein the map transition policy indicates a transition from the interim processing node map to the final processing map based on a threshold amount of time.
20 . The non-transitory computer-readable media of claim 18 , wherein the map transition policy indicates a transition from the interim processing node map to the final processing map based on a quantity of cache misses.