Efficient I/O error analysis and proactive I/O failover to alternate paths for InfiniBand channel
An InfiniBand managed storage environment is made up of processor nodes containing HCAs and managed storage devices containing TCAs and exposing a plurality of LUNs and volumes. For each InfiniBand channel between a specific HCA and a specific TCA, the paths between the HCA and any LUN or volume exposed by the TCA are grouped into a set. Occurrence of failures on specific paths of specific sets on specific channels are determined, for example by registering for callbacks or polling for occurrence of events which adversely affect communication between endpoints. Also, I/O operations executed by processor nodes are tracked and failures thereof are detected. When the occurrence of a failure on a specific path of a set is determined, all I/O operations on all paths of the set are proactively rerouted to a separate set on a separate channel that connects the same processor node and storage device.
1. A computer implemented method for proactively rerouting input/output (“I/O”) operations in an InfiniBand managed storage environment comprising at least one processor node containing a host channel adapter (“HCA”) and at least one managed storage device containing at least one target channel adapter (“TCA”) exposing a plurality of logical units (“LUNs”) and logical volumes, the method comprising the steps of:
for each InfiniBand channel between a specific HCA and a specific TCA in the InfiniBand managed storage environment, grouping, by a computer, each one of a plurality of I/O paths between the HCA and any LUN or logical volume exposed by the TCA into a path set;
determining, by the computer, an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel; and
proactively rerouting all I/O operations on all paths of the specific path set to paths of a separate path set on a separate InfiniBand channel, without waiting for I/O errors to occur on other paths of the specific path set, wherein the specific InfiniBand channel and the separate InfiniBand channel both connect a specific processor node to a specific managed storage device.
2. The method of claim 1 further comprising:
registering a callback for each of a plurality of InfiniBand events which adversely affect communication between endpoints in the InfiniBand managed storage environment; and
wherein determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel further comprises receiving a callback indicating occurrence of an event for which a callback was registered, wherein the occurrence of the event is on the specific path of the specific path set.
3. The method of claim 2 wherein registering a callback for an InfiniBand event further comprises:
registering a callback for an InfiniBand event with an InfiniBand hardware component.
4. The method of claim 2 wherein registering a callback for an InfiniBand event further comprises:
registering a callback for an InfiniBand event with an InfiniBand software component.
5. The method of claim 2 wherein registering a callback for an InfiniBand event further comprises:
registering a callback for at least one event from a group of events consisting of: a channel adapter failure, a connection failure, a port failure, a switch failure and a link failure.
6. The method of claim 1 further comprising:
polling at given intervals to determine occurrence of each of a plurality of InfiniBand events which adversely affect communication between endpoints in the InfiniBand managed storage environment; and
wherein determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel further comprises learning of an occurrence of an event on the specific path of the specific path set, as a result of polling to determine occurrence of the event.
7. The method of claim 6 wherein polling to determine occurrence of an InfiniBand event further comprises:
polling an InfiniBand hardware component to determine occurrence of an InfiniBand event.
8. The method of claim 6 wherein polling to determine occurrence of an InfiniBand event further comprises:
polling an InfiniBand software component to determine occurrence of an InfiniBand event.
9. The method of claim 6 wherein polling to determine occurrence of an InfiniBand event further comprises:
polling to determine occurrence of at least one event from a group of events consisting of: a channel adapter failure, a connection failure, a port failure, a switch failure and a link failure.
10. The method of claim 1 further comprising:
tracking I/O operations executed by processor nodes of the InfiniBand managed storage environment; and
wherein determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel further comprises detecting failure of a specific I/O operation on the specific path of the specific path set.
11. The method of claim 1 further comprising:
maintaining at least two separate InfiniBand channels between at least one specific processor node and at least one specific managed storage device; and
wherein proactively rerouting all I/O operations on all paths of the specific path set to paths of the separate path set further comprises rerouting the I/O operations from a path set on a first one of the at least two separate InfiniBand channels between the specific processor node and the specific managed storage device to a path set on a second one of the at least two separate InfiniBand channels between the specific processor node and the specific managed storage device.
12. The method of claim 1 further comprising:
subsequently to rerouting all I/O operations on all paths of the specific path set to paths of the separate path set, verifying correction of the determined failure; and
rerouting I/O operations on the separate path set back to their original path set.
13. At least one non-transitory computer readable-storage medium for proactively rerouting input/output (“I/O”) operations in an InfiniBand managed storage environment comprising at least one processor node containing a host channel adapter (“HCA”) and at least one managed storage device containing at least one target channel adapter (“TCA”) exposing a plurality of logical units (“LUNs”) and logical volumes, the at least one non-transitory computer readable-storage medium storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following steps:
for each InfiniBand channel between a specific HCA and a specific TCA in the InfiniBand managed storage environment, grouping each one of a plurality of I/O paths between the HCA and any LUN or logical volume exposed by the TCA into a path set;
determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel; and
proactively rerouting all I/O operations on all paths of the specific path set to paths of a separate path set on a separate InfiniBand channel, without waiting for I/O errors to occur on other paths of the specific path set, wherein the specific InfiniBand channel and the separate InfiniBand channel both connect a specific processor node to a specific managed storage device.
14. The at least one non-transitory computer readable-storage medium of claim 13 further storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following step:
registering a callback for each of a plurality of InfiniBand events which adversely affect communication between endpoints in the InfiniBand managed storage environment; and
wherein determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel further comprises receiving a callback indicating occurrence of an event for which a callback was registered, wherein the occurrence of the event is on the specific path of the specific path set.
15. The at least one non-transitory computer readable-storage medium of claim 14 wherein registering a callback for an InfiniBand event further comprises:
registering a callback for an InfiniBand event with an InfiniBand hardware component.
16. The at least one non-transitory computer readable-storage medium of claim 14 wherein registering a callback for an InfiniBand event further comprises:
registering a callback for an InfiniBand event with an InfiniBand software component.
17. The at least one non-transitory computer readable-storage medium of claim 13 further storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following step:
polling at given intervals to determine occurrence of each of a plurality of InfiniBand events which adversely affect communication between endpoints in the InfiniBand managed storage environment; and
wherein determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel further comprises learning of an occurrence of an event on the specific path of the specific path set, as a result of polling to determine occurrence of the event.
18. The at least one non-transitory computer readable-storage medium of claim 13 further storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following step:
tracking I/O operations executed by processor nodes of the InfiniBand managed storage environment; and
wherein determining an occurrence of a failure on a specific path of a specific path set on a specific InfiniBand channel further comprises detecting failure of a specific I/O operation on the specific path of the specific path set.
19. The at least one non-transitory computer readable-storage medium of claim 13 further storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following step:
maintaining at least two separate InfiniBand channels between at least one specific processor node and at least one specific managed storage device; and
wherein proactively rerouting all I/O operations on all paths of the specific path set to paths of the separate path set further comprises rerouting the I/O operations from a path set on a first one of the at least two separate InfiniBand channels between the specific processor node and the specific managed storage device to a path set on a second one of the at least two separate InfiniBand channels between the specific processor node and the specific managed storage device.
20. A computer system for proactively rerouting input/output (“I/O”) operations in an InfiniBand managed storage environment comprising at least one processor node containing a host channel adapter (“HCA”) and at least one managed storage device containing at least one target channel adapter (“TCA”) exposing a plurality of logical units (“LUNs”) and logical volumes, the computer system comprising:
at least one processor;
system memory;
a path grouping module residing in the system memory, the path grouping module being programmed to group, for each InfiniBand channel between a specific HCA and a specific TCA in the InfiniBand managed storage environment, each one of a plurality of I/O paths between the HCA and any LUN or logical volume exposed by the TCA into a path set;
a callback registering module residing in the system memory, the callback registering module being programmed to register a callback for each of a plurality of InfiniBand events which adversely affect communication between endpoints in the InfiniBand managed storage environment;
an event polling module residing in the system memory, the event polling module being programmed to poll at given intervals to determine occurrence of each of a plurality of InfiniBand events which adversely affect communication between endpoints in the InfiniBand managed storage environment;
an I/O operation tracking module residing in the system memory, the I/O operation tracking module being programmed to track I/O operations executed by processor nodes of the InfiniBand managed storage environment; and
a path set rerouting module residing in the system memory, the path set rerouting module being programmed to reroute all I/O operations on all paths of the specific path set to paths of a separate path set on a separate InfiniBand channel proactively, without waiting for I/O errors to occur on other paths of the specific path set, wherein the specific InfiniBand channel and the separate InfiniBand channel both connect a specific processor node to a specific managed storage device.