Cluster group change preparation techniques
Described herein are technologies directed to cluster group change preparation techniques. Prior to a cluster group change operation, such as a merge or a split of a node with a group, a notification of a planned group change operation can be sent to the nodes of a group. In response to the notification, the nodes of the group can perform group change preparation operations disclosed herein. The group change operation can be blocked while the group change preparation operations are underway. The group change operation can proceed after the nodes of the group indicate completion of the group change preparation operations.
1. A method, comprising:
receiving, by a first node device in a group of node devices, a notification of a planned merge operation wherein a second node device is to merge into the group of node devices;
initially indicating, by the first node device, that the planned merge operation is blocked;
while the planned merge operation is blocked, performing, by the first node device, at least one group change preparation operation, wherein the at least one group change preparation operation comprises draining a cached lazy lock queue;
in response to completing the at least one group change preparation operation by the first node device, subsequently indicating, by the first node device, that the planned merge operation is unblocked; and
initiating, by the first node device, the planned merge operation to merge the second node device into the group of node devices.
2. The method of claim 1 , wherein the first node device, the second node device, and the group of node devices are in a cluster of node devices.
3. The method of claim 1 , wherein at least the notification of the planned merge operation is performed according to a drain service of a group management protocol.
4. The method of claim 1 , wherein multiple respective node devices within the group of node devices indicate that the planned merge operation is unblocked in response to completing respective group change preparation operations at the multiple respective node devices.
5. The method of claim 4 , wherein the initiating the planned merge operation to merge the second node device into the group of node devices commences in response to the multiple respective node devices indicating that the planned merge operation is unblocked.
6. The method of claim 1 , wherein the cached lazy lock queue comprises a cached queue of locks held for the first node device.
7. The method of claim 6 , wherein the planned merge operation to merge the second node device into the group of node devices comprises reselecting, by the first node device, one or more other node devices in the group of node devices, other than the first node device, to serve as lock coordinators for the locks held for the first node device.
8. A node device in a cluster of node devices, comprising:
at least one processor; and
at least one memory that stores executable instructions that, when executed by the at least one processor, facilitate performance of operations, comprising:
notifying other node devices in the cluster of node devices, other than the node device, of a planned split operation, wherein the node device is to split from a group of node devices within the cluster of node devices;
in response to the planned split operation, performing at least one group change preparation operation by the node device, wherein the at least one group change preparation operation comprises draining a cached lazy lock queue; and
in response to completion of the at least one group change preparation operation by the node device and the other node devices within the group of node devices, or in response to elapse of a time interval, initiating, by the node device, the planned split operation.
9. The node device of claim 8 , wherein the notifying the other node devices in the cluster of node devices of the planned split operation is performed in response to a shutdown command received at the node device.
10. The node device of claim 8 , wherein at least the notifying the other node devices in the cluster of node devices of the planned split operation is performed according to a drain service of a group management protocol.
11. The node device of claim 8 , wherein a respective node device within the group of node devices performs a respective group change preparation operation at the respective node device, in response to receiving a notification of the planned split operation.
12. The node device of claim 8 , wherein the cached lazy lock queue comprises a cached queue of locks held for the node device.
13. The node device of claim 8 , wherein the initiating the planned split operation commences after each respective node device of the other node devices within the group of node devices has completed a respective group change preparation operation.
14. The node device of claim 8 , wherein the planned split operation to split the node device from the group of node devices comprises reselecting, by the other node devices within the group of node devices, ones of the node devices in the group of node devices to serve as lock coordinators for locks held for the other node devices within the group of node devices.
15. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
receiving a notification of a planned group change operation, wherein the group change operation comprises a merge of a node device with a group of node devices within a cluster of node devices, or a split of the node device from the group of node devices within the cluster of node devices;
in response to the receiving of the notification of the planned group change operation, notifying the group of node devices of the planned group change operation; and
after each respective node device of the group of node devices has completed a respective group change preparation operation, initiating the planned group change operation by at least one node device within the group of node devices,
wherein at least one respective group change preparation operation completed by a respective node device of the group of node devices comprises draining a cached lazy lock queue.
16. The non-transitory machine-readable medium of claim 15 , wherein the notification of the planned group change operation is performed according to a drain service of a group management protocol.
17. The non-transitory machine-readable medium of claim 15 , wherein the cached lazy lock queue comprises a cached queue of locks held for the respective node device.
18. The non-transitory machine-readable medium of claim 15 , wherein the planned group change operation comprises the merge of the node device with the group of node devices within the cluster of node devices, and wherein the operations further comprise determining that each respective node device of the group of node devices has completed the respective group change preparation operation by observing releases of respective merge blocked flags for respective node devices of the group of node devices.
19. The non-transitory machine-readable medium of claim 15 , wherein the planned group change operation comprises the split of the node device from the group of node devices within the cluster of node devices, and wherein the operations further comprise determining that each respective node device of the group of node devices has completed the respective group change preparation operation by observing whether a drain goal is met at respective node devices of the group of node devices.
20. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise measuring a time interval beginning after notifying the group of node devices of the planned group change operation.