Enhanced fencing scheme for cluster systems without inherent hardware fencing
Techniques for providing a fencing scheme for cluster systems without inherent hardware fencing. Storage nodes in an HA cluster communicate with one another over communication paths implemented through a variety of mechanisms, including drives and network connections. Each storage node in the HA cluster executes a fencing enforcer component operable to enforce actions or processes for initiating fencing of itself and/or initiating self-fencing of another storage node in the HA cluster determined to be malfunctioning. By providing for richer communications between storage nodes in an HA cluster and a richer set of actions or processes for fencing a malfunctioning storage node including self-fencing, the malfunctioning storage node can be made to exit the HA cluster in a more controlled fashion. In addition, a remaining storage node in the HA cluster can more safely take on the role of primary storage node with reduced risk of data corruption on shared resources.
1. A cluster system, comprising:
a first storage node and a second storage node arranged in an active-active storage system configuration,
wherein the first storage node is configured as a first virtual machine (VM) executing on at least one hypervisor host, and
wherein the second storage node is configured as a second VM executing on the at least one hypervisor host; and
a plurality of storage drives communicably coupled to the first VM and the second VM,
wherein the first VM is configured to:
engage in communications with the second VM through a plurality of communication mechanisms including the plurality of storage drives and multiple network or channel connections;
determine that the second VM is malfunctioning based at least on a response or a lack of response to the communications;
enforce one or more actions or processes to initiate self-fencing of the second VM determined to be malfunctioning;
obtain, from the at least one hypervisor host, an identifier (ID) of the second VM determined to be malfunctioning; and
force a kernel panic at the second VM having the obtained ID by issuing a non-maskable interrupt (NMI) on the at least one hypervisor host.
2. The cluster system of claim 1 wherein the first VM is further configured to:
fail to receive a notification of the kernel panic; and
having failed to receive the notification of the kernel panic, send a request to the at least one hypervisor host to terminate and restart the second VM having the obtained ID.
3. The cluster system of claim 1 wherein the second VM is configured to send a notification to inform the first VM of the kernel panic.
4. The cluster system of claim 3 wherein the second VM is further configured to execute a kernel routine to output an error message.
5. The cluster system of claim 4 wherein the second VM is further configured to dump an image of kernel memory to a respective storage drive from among the plurality of storage drives for subsequent analysis and debugging.
6. The cluster system of claim 5 wherein the second VM is further configured either to wait to be restarted or to initiate an automatic restart.
7. A method, comprising:
engaging in communications by a first storage node with a second storage node through a plurality of communication mechanisms including a plurality of storage drives and multiple network or channel connections, the first storage node and the second storage node being arranged in an active-active storage system configuration, the first storage node being configured as a first virtual machine (VM) executing on at least one hypervisor host, the second storage node being configured as a second VM executing on the at least one hypervisor host, the plurality of storage drives being communicably coupled to the first VM and the second VM;
determining, by the first VM, that the second VM is malfunctioning based at least on a response or a lack of response to the communications;
enforcing, by the first VM, one or more actions or processes to initiate self-fencing of the second VM determined to be malfunctioning;
obtaining, by the first VM from the at least one hypervisor host, an identifier (ID) of the second VM determined to be malfunctioning; and
forcing, by the first VM, a kernel panic at the second VM having the obtained ID by issuing a non-maskable interrupt (NMI) on the at least one hypervisor host.
8. The method of claim 7 further comprising:
failing to receive, at the first VM, a notification of the kernel panic; and
having failed to receive the notification of the kernel panic, sending, by the first VM, a request to the at least one hypervisor host to terminate and restart the second VM having the obtained ID.
9. The method of claim 7 further comprising:
sending, by the second VM, a notification to inform the first VM of the kernel panic.
10. The method of claim 9 further comprising:
executing, by the second VM, a kernel routine to output an error message.
11. The method of claim 10 further comprising:
dumping, by the second VM, an image of kernel memory to a respective storage drive from among the plurality of storage drives for subsequent analysis and debugging.
12. The method of claim 11 further comprising:
performing, by the second VM, one of waiting to be restarted and initiating an automatic restart.
13. A computer program product including a set of non-transitory, computer-readable media having program instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:
engaging in communications by a first storage node with a second storage node through a plurality of communication mechanisms including a plurality of storage drives and multiple network or channel connections, the first storage node and the second storage node being arranged in an active-active storage system configuration, the first storage node being configured as a first virtual machine (VM) executing on at least one hypervisor host, the second storage node being configured as a second VM executing on the at least one hypervisor host, the plurality of storage drives being communicably coupled to the first VM and the second VM;
determining, by the first VM, that the second VM is malfunctioning based at least on a response or a lack of response to the communications;
enforcing, by the first VM, one or more actions or processes to initiate self-fencing of the second VM determined to be malfunctioning;
obtaining, by the first VM from the at least one hypervisor host, an identifier (ID) of the second VM determined to be malfunctioning; and
forcing, by the first VM, a kernel panic at the second VM having the obtained ID by issuing a non-maskable interrupt (NMI) on the at least one hypervisor host.
14. The computer program product of claim 13 wherein the method further comprises:
failing to receive, at the first VM, a notification of the kernel panic; and
having failed to receive the notification of the kernel panic, sending, by the first VM, a request to the at least one hypervisor host to terminate and restart the second VM having the obtained ID.
15. The computer program product of claim 13 wherein the method further comprises:
sending, by the second VM, a notification to inform the first VM of the kernel panic.
16. The computer program product of claim 15 wherein the method further comprises:
executing, by the second VM, a kernel routine to output an error message.
17. The computer program product of claim 16 wherein the method further comprises:
dumping, by the second VM, an image of kernel memory to a respective storage drive from among the plurality of storage drives for subsequent analysis and debugging.
18. The computer program product of claim 17 wherein the method further comprises:
performing, by the second VM, one of waiting to be restarted and initiating an automatic restart.