System and method for customized I/O fencing for preventing data corruption in computer system clusters
Systems, methods, apparatus and software can implement a flexible I/O fence mechanism framework allowing clustered computer systems to conveniently use one or more I/O fencing techniques. Various different fencing techniques can be used, and fencing mechanism can be customized.
1. A method, performed on a first node of a plurality of nodes, said method comprising:
detecting when a computer system cluster including said plurality of nodes is partitioned, wherein
a plurality of I/O fence mechanisms resides on the first node of said plurality of nodes;
configuring said computer system cluster such that a first one of the plurality of I/O fence mechanisms takes priority over a second one of the at least two of the plurality of I/O fence mechanisms;
attempting to gain control of at least one coordinator resource using the first one of the plurality of I/O fence mechanisms; and
removing at least one of the plurality of nodes from the computer system cluster in response to the attempting to gain control.
2. The method of claim 1 further comprising: generating fence mechanism comparison information corresponding to the first one of the plurality of I/O fence mechanisms, wherein the first one of the at least one of the plurality of I/O fence mechanisms is configured to operate on the first node of the plurality of nodes; and
comparing the fence mechanism comparison information with at least one of: remote fence mechanism comparison information corresponding to a remote fence mechanism configured to operate on a second one of the plurality of nodes, and
local fence mechanism comparison information stored on the first one of the plurality of nodes.
3. The method of claim 1 wherein
the at least one of the plurality of I/O fence mechanisms uses the at least one coordinator resource, and
the at least one coordinator resource includes at least one of:
a shared data resource supporting small computer system interface version three (SCSI-3) persistent reservation commands,
a shared data resource supporting SCSI-2 reservation commands;
a storage area network (SAN) switch;
a power distribution unit coupled to one of the plurality of nodes; an Intelligent Platform Management Interface (IPMI) device;
a management processor coupled to the plurality of nodes;
a process operating on a computer system;
a volume; and
a contact-response system.
4. The method of claim 1 , further comprising:
attempting to gain control of a second one of the at least one coordinator resource using the second one of the at least two one of the plurality of I/O fence mechanisms.
5. The method of claim 1 further comprising:
joining at least one of the plurality of nodes to the computer system cluster, wherein the manner of joining depends on the plurality of I/O fence mechanisms.
6. The method of claim 1 wherein the attempting further comprises:
sending a command to the at least one coordinator resource, the command for causing a change in power state for at least one of the plurality of nodes.
7. The method of claim 1 wherein the attempting further comprises:
sending a command to the at least one coordinator resource, the command for unregistering at least another one of the plurality of nodes from the at least one coordinator resource.
8. The method of claim 1 wherein the detecting further comprises:
monitoring a communications link between each of the plurality of nodes for a heartbeat signal; and
determining when the heartbeat signal is not present for a specified period of time.
9. The method of claim 1 further comprising:
sending, when the attempting is successful, a message to at least another one of the plurality of nodes that the attempting was successful.
10. A system comprising:
a first node including:
a first processor;
a first memory coupled to the first processor;
a plurality of fence mechanism modules residing on said first node, wherein each fence mechanism module of the plurality of fence mechanism modules is configured to implement at least one input/output (I/O) fence mechanism;
a fence mechanism manager residing on said first node and configured to invoke at least one of the plurality of fence mechanism modules, wherein
the fence mechanism manager is configured to select a first one of the plurality of fence mechanism modules to take priority over a second one of the plurality of fence mechanism modules; and
a fence driver residing on said first node, in communication with the fence mechanism manager and configured to:
receive an indication that a distributed computer system has partitioned into at least two subclusters, and
attempt to control at least one coordinator resource using a first one of the at least one I/O fence mechanism.
11. The system of claim 10 wherein at least one of the fence mechanism manager, the at least one of the plurality of fence mechanism modules, and the fence driver is further configured to:
generate fence mechanism comparison information corresponding to at least one of the plurality of fence mechanism modules.
12. The system of claim 11 wherein at least one of the fence mechanism manager, the at least one of the plurality of fence mechanism modules, and the fence driver is further configured to:
compare the fence mechanism comparison information with at least one of:
remote fence mechanism comparison information and local fence mechanism comparison information.
13. The system of claim 10 wherein the at least one coordinator resource includes at least one of: a shared data resource supporting small computer system interface version three (SCSI-3) persistent reservation commands, a shared data resource supporting SCSI-2 reservation commands; a storage area network (SAN) switch; a power distribution unit coupled to one of the plurality of nodes; an Intelligent Platform Management Interface (IPMI) device; a management processor coupled to the plurality of nodes; a process operating on a computer system; a volume; and a contact-response system.
14. The system of claim 10 ,
wherein at least one of the fence mechanism manager, the plurality of fence mechanism modules, and the fence driver are encoded as instructions stored in at least one of the first memory and the second memory and executable on at least one of the first processor and the second processor.
15. The system of claim 14 wherein a first portion of the first memory is used as a kernel memory space and wherein a second portion of the first memory is used as a user memory space, and wherein the fence driver is stored in the kernel memory space.
16. The system of claim 10 wherein at least one of the fence mechanism manager, the plurality of fence mechanism modules, and the fence driver is further configured to:
determine which of a plurality of nodes in one of the at least two subclusters will race for control of the at least one coordinator resource.
17. The system of claim 10 wherein the plurality of fence mechanism modules is further configured to:
join at least one of a plurality of nodes to the distributed computer system, wherein the manner of joining depends on the at least one I/O fence mechanism implemented by the plurality of fence mechanism modules.
18. The system of claim 10 wherein the plurality of fence mechanism modules is further configured to:
send a command to the at least one coordinator resource, the command for causing a change in power state for at least one of a plurality of nodes.
19. The system of claim 10 wherein the plurality of fence mechanism modules is further configured to:
send a command to the at least one coordinator resource, the command for unregistering at least one of a plurality of nodes from the at least one coordinator resource.
20. A non-transitory computer readable storage medium comprising program instructions executable on a processor, wherein the program instructions are operable to cause said processor to implement each of:
detecting when a computer system cluster including a plurality of nodes is partitioned, wherein
a plurality of I/O fence mechanisms resides on a first node of said plurality of nodes;
configuring said computer system cluster such that a first one of the at least one of the plurality of I/O fence mechanisms takes priority over a second one of the plurality of I/O fence mechanisms;
attempting to gain control of at least one coordinator resource using the first one of the plurality of I/O fence mechanisms; and
removing at least one of the plurality of nodes from the computer system cluster in response to the attempting to gain control.
21. The non-transitory computer readable storage medium of claim 20 further comprising program instructions operable to implement each of:
generating fence mechanism comparison information corresponding to the plurality of I/O fence mechanisms, wherein
the first one of the plurality of I/O fence mechanisms is configured to operate on the first node of the plurality of nodes; and
comparing the fence mechanism comparison information with at least one of:
remote fence mechanism comparison information corresponding to a fence mechanism configured to operate on a second one of the plurality of nodes, and local fence mechanism comparison information stored on the first one of the plurality of nodes.
22. The non-transitory computer readable storage medium of claim 20 wherein the at least one coordinator resource includes at least one of: a shared data resource supporting small computer system interface version three (SCSI-3) persistent reservation commands, a shared data resource supporting SCSI-2 reservation commands; a storage area network (SAN) switch; a power distribution unit coupled to one of the plurality of nodes; an Intelligent Platform Management Interface (IPMI) device; a management processor coupled to the plurality of nodes; a process operating on a computer system; a volume; and a contact-response system.
23. The non-transitory computer readable storage medium of claim 20 wherein the selecting at least one of a plurality of I/O fence mechanisms further comprises program instructions operable to implement selecting at least two of the plurality of I/O fence mechanisms, and wherein the attempting to gain control of the at least one coordinator resource using the at least one of a plurality of I/O fence mechanisms further comprises program instructions operable to implement each of:
attempting to gain control of a first one of the at least one coordinator resource using a first one of the at least two of the plurality of I/O fence mechanisms; and
attempting to gain control of a second one of the at least one coordinator resource using a second one of the at least two of the plurality of I/O fence mechanisms.
24. The non-transitory computer readable storage medium of claim 20 further comprising program instructions operable to implement:
joining at least one of the plurality of nodes to the computer system cluster, wherein the manner of joining depends on the selecting at least one of a plurality of I/O fence mechanisms.
25. The non-transitory computer readable storage medium of claim 20 further comprising program instructions operable to implement:
sending a command to the at least one coordinator resource, the command for causing a change in power state for at least one of the plurality of nodes.
26. The non-transitory computer readable storage medium of claim 20 further comprising program instructions operable to implement:
sending a command to the at least one coordinator resource, the command for unregistering at least another one of the plurality of nodes from the at least one coordinator resource.
27. The non-transitory computer readable storage medium of claim 20 further comprising program instructions operable to implement each of:
monitoring a communications link between each of the plurality of nodes for a heartbeat signal; and
determining when the heartbeat signal is not present for a specified period of time.
28. The non-transitory computer readable storage medium of claim 20 further comprising program instructions operable to implement:
sending, when the attempting is successful, a message to at least another one of the plurality of nodes that the attempting was successful.
29. An apparatus comprising:
a processor;
a means for causing said processor to perform detecting when a computer system cluster including a plurality of nodes is partitioned, wherein
a plurality of I/O fence mechanisms resides on a first node of said plurality of nodes;
a means for configuring said computer system cluster such that a first one of the plurality of I/O fence mechanisms takes priority over a second one of the plurality of I/O fence mechanisms;
a means for causing said processor to perform attempting to gain control of at least one coordinator means using the first one of the plurality of I/O fence mechanisms; and
a means for causing said processor to perform removing at least one of the plurality of nodes from the computer system cluster.
30. The apparatus of claim 29 further comprising:
a means for generating I/O fencing comparison information corresponding to the at least first one of the plurality of I/O fence mechanisms;
a means for comparing the I/O fencing comparison information with at least one of: remote comparison information, and local comparison information.
31. The apparatus of claim 29 further comprising:
a means for joining at least one of the plurality of nodes to the computer system cluster, wherein the manner of joining depends on the plurality of I/O fence mechanisms.
32. The apparatus of claim 29 further comprising:
a means for sending a command to the at least one coordinator means, the command for causing a change in power state for at least one of the plurality of nodes.
33. The apparatus of claim 29 further comprising:
a means for sending a message to at least another one of the plurality of nodes when that the means for attempting to gain control of the at least one coordinator means is successful.