Reassigning a processing node from downloading to searching a data group
A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.
1. A method, comprising:
identifying a plurality of processing nodes;
assigning a first processing node of the plurality of processing nodes to search a first data group, wherein based on assigning the first processing node to search the first data group, the first processing node is configured to execute searches on at least a portion of the first data group;
assigning the first processing node to download a second data group, wherein a second processing node of the plurality of processing nodes is assigned to search the second data group, and wherein based on assigning the first processing node to download the second data group, the first processing node is configured to download at least a portion of the second data group from a shared storage system, wherein the at least a portion of the second data group is at least one of generated by the second processing node, stored to the shared storage system by the second processing node, or searched by the second processing node; and
based on an assignment transition policy, reassigning searching of the second data group from the second processing node to the first processing node, wherein based on reassigning searching of the second data group from the second processing node to the first processing node, the first processing node is configured to execute searches on the at least a portion of the second data group.
2. The method of claim 1 , wherein the plurality of processing nodes is a plurality of distributed processing nodes.
3. The method of claim 1 , wherein each data group of a plurality of data groups is assigned to a respective processing node of the plurality of processing nodes.
4. The method of claim 1 , further comprising assigning the first processing node to download a third data group and a fourth data group, wherein a third processing node of the plurality of processing nodes is assigned to search the third data group and a fourth processing node of the plurality of processing nodes is assigned to search the fourth data group.
5. The method of claim 1 , wherein a third processing node of the plurality of processing nodes is assigned to download the first data group, the method further comprising assigning the third processing node to download the second data group.
6. The method of claim 1 , wherein a third processing node of the plurality of processing nodes is assigned to download the first data group, the method further comprising assigning a fourth processing node of the plurality of processing nodes to download the second data group.
7. The method of claim 1 , wherein the first data group is a first partition, the first partition comprising a first plurality of buckets, wherein the second data group is a second partition, the second partition comprising a second plurality of buckets.
8. The method of claim 1 , wherein based on assigning the first processing node to download the second data group, the first processing node is further configured to copy the at least a portion of the second data group.
9. The method of claim 1 , wherein based on reassigning the searching of the second data group from the second processing node to the first processing node, the second processing node is not configured to execute searches on the at least a portion of the second data group.
10. The method of claim 1 , further comprising:
determining that each processing node of the plurality of processing nodes is activated; and
determining that each processing node of the plurality of processing nodes is available to execute one or more queries.
11. The method of claim 1 , wherein the assignment transition policy indicates that reassignment of the searching of the second data group from the second processing node to the first processing node is based on a time period.
12. The method of claim 1 , wherein the assignment transition policy indicates that reassignment of the searching of the second data group from the second processing node to the first processing node is based on a number of caches misses by the first processing node.
13. The method of claim 1 , wherein the assignment transition policy indicates that reassignment of the searching of the second data group from the second processing node to the first processing node is based on a number of searches executed by the first processing node.
14. The method of claim 1 , wherein, based on the assignment transition policy, a third processing node of the plurality of processing nodes is assigned to search the second data group.
15. The method of claim 1 , further comprising, based on the assignment transition policy, reassigning downloading of the second data group from the first processing node to the second processing node.
16. The method of claim 1 , wherein assigning the first processing node to search the first data group comprises assigning the first processing node to search the first data group based on identifying the plurality of processing nodes.
17. The method of claim 1 , further comprising determining the plurality of processing nodes comprises the first processing node, wherein assigning the first processing node to search the first data group comprises assigning the first processing node to search the first data group based on determining the plurality of processing nodes comprises the first processing node.
18. The method of claim 1 , wherein the shared storage system is accessible by each processing node of the plurality of processing nodes.
19. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processors coupled to the memory and configured to:
identify a plurality of processing nodes;
assign a first processing node of the plurality of processing nodes to search a first data group, wherein based on assigning the first processing node to search the first data group, the first processing node is configured to execute searches on at least a portion of the first data group;
assign the first processing node to download a second data group, wherein a second processing node of the plurality of processing nodes is assigned to search the second data group, and wherein based on assigning the first processing node to download the second data group, the first processing node is configured to download at least a portion of the second data group from a shared storage system, wherein the at least a portion of the second data group is at least one of generated by the second processing node, stored to the shared storage system by the second processing node, or searched by the second processing node; and
based on an assignment transition policy, reassign searching of the second data group from the second processing node to the first processing node, wherein based on reassigning searching of the second data group from the second processing node to the first processing node, the first processing node is configured to execute searches on the at least a portion of the second data group.
20. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
identify a plurality of processing nodes;
assign a first processing node of the plurality of processing nodes to search a first data group, wherein based on assigning the first processing node to search the first data group, the first processing node is configured to execute searches on at least a portion of the first data group;
assign the first processing node to download a second data group, wherein a second processing node of the plurality of processing nodes is assigned to search the second data group, and wherein based on assigning the first processing node to download the second data group, the first processing node is configured to download at least a portion of the second data group from a shared storage system, wherein the at least a portion of the second data group is at least one of generated by the second processing node, stored to the shared storage system by the second processing node, or searched by the second processing node; and
based on an assignment transition policy, reassign searching of the second data group from the second processing node to the first processing node, wherein based on reassigning searching of the second data group from the second processing node to the first processing node, the first processing node is configured to execute searches on the at least a portion of the second data group.