Multi-host environment resiliency
One or more aspects of the present disclosure relate to achieving resiliency and high availability in a multi-host environment. In embodiments, a plurality of Peripheral Component Interconnect Express (PCIe) links and network interface cards (NICs) controlled by a plurality of hosts in a multi-host environment is monitored. In addition, a fault condition is detected. For example, the fault condition includes at least an unexpected NIC reset, a PCIe link fault, or a server power cycle event. Further, traffic transmission over a network is controlled based on the fault condition.
1 . A method comprising:
monitoring, by a fabric monitoring subsystem, a plurality of Peripheral Component Interconnect Express (PCIe) links and network interface cards (NICs) controlled by a plurality of hosts in a multi-host environment;
detecting, by the fabric monitoring subsystem, a fault condition, wherein the fault condition includes at least an unexpected NIC reset, a PCIe link fault, or a server power cycle event, wherein detecting the fault condition comprises differentiating between the unexpected NIC reset, the PCIe link fault, and the server power cycle event using signature analysis techniques that compare observed network behaviors against known fault signatures stored in memory; and
controlling traffic transmission over a network based on the fault condition, wherein controlling traffic transmission comprises selecting an optimal traffic routing path that minimizes latency and maximizes throughput based on bandwidth, latency, and error rate criteria.
2 . The method of claim 1 , further comprising:
transmitting a fault notification to a subject host of the plurality of hosts affected by the fault condition.
3 . The method of claim 2 , further comprising:
initiating a traffic path failover process in response to the fault notification.
4 . The method of claim 1 , further comprising:
disabling data traffic paths corresponding to the fault condition.
5 . The method of claim 1 , further comprising:
continuing network operations over data traffic paths alternate to those data traffic paths corresponding to the fault condition.
6 . The method of claim 1 , further comprising:
notifying one or more endpoints on the network and in the multi-host environment of the fault condition.
7 . The method of claim 6 , further comprising:
using an out-of-band control messaging interface to notify and send control messages to a subject host of the plurality of hosts affected by the fault condition.
8 . The method of claim 1 , further comprising:
transmitting a hardware interrupt indicative of the fault condition to a subject host of the plurality of hosts affected by the fault condition, wherein the hardware interrupt is a non-maskable interrupt (NMI) signaling a non-recoverable hardware error, and wherein a distinct interrupt signal is generated based on the type of fault condition to differentiate the unexpected NIC reset from the PCIe link fault.
9 . The method of claim 8 , further comprising:
determining, by the subject host, whether the hardware interrupt signals a NIC reset or a PCIe link fault based on the distinct interrupt signal;
executing, by the subject host, a rapid cleanup of in-flight traffic through each PCIe link and each NIC corresponding to the fault condition, wherein executing the rapid cleanup comprises: identifying all input/output (IO) operations associated with a source of a fault notification corresponding to the fault condition; and issuing abort signals to terminate the identified IO operations to prevent data loss and system disruption;
disabling, by the subject host, data traffic paths through each PCIe link and each NIC corresponding to the fault condition, wherein if the fault notification corresponds to a NIC reset, all paths through a faulted NIC are disabled, and if the fault notification corresponds to a PCIe link fault, only paths associated with the PCIe link fault are disabled.
10 . The method of claim 9 , further comprising:
maintaining, by the subject host, network operations through an operational NIC or PCIe link.
11 . An apparatus with a memory and processor, the apparatus configured to:
monitor, by a fabric monitoring subsystem, a plurality of Peripheral Component Interconnect Express (PCIe) links and network interface cards (NICs) controlled by a plurality of hosts in a multi-host environment;
detect, by the fabric monitoring subsystem, a fault condition, wherein the fault condition includes at least an unexpected NIC reset, a PCIe link fault, or a server power cycle event, wherein detecting the fault condition comprises differentiating between the unexpected NIC reset, the PCIe link fault, and the server power cycle event using signature analysis techniques that compare observed network behaviors against known fault signatures stored in memory; and
control traffic transmission over a network based on the fault condition, wherein controlling traffic transmission comprises selecting an optimal traffic routing path that minimizes latency and maximizes throughput based on bandwidth, latency, and error rate criteria.
12 . The apparatus of claim 11 , further configured to:
transmit a fault notification to a subject host of the plurality of hosts affected by the fault condition.
13 . The apparatus of claim 12 , further configured to:
initiate a traffic path failover process in response to the fault notification.
14 . The apparatus of claim 11 , further configured to:
disable data traffic paths corresponding to the fault condition.
15 . The apparatus of claim 11 , further configured to:
continue network operations over data traffic paths alternate to those data traffic paths corresponding to the fault condition.
16 . The apparatus of claim 11 , further configured to:
notify one or more endpoints on the network and in the multi-host environment of the fault condition.
17 . The apparatus of claim 16 , further configured to:
use an out-of-band control messaging interface to notify and send control messages to a subject host of the plurality of hosts affected by the fault condition.
18 . The apparatus of claim 11 , further configured to:
transmit a hardware interrupt indicative of the fault condition to a subject host of the plurality of hosts affected by the fault condition, wherein the hardware interrupt is a non-maskable interrupt (NMI) signaling a non-recoverable hardware error, and wherein a distinct interrupt signal is generated based on the type of fault condition to differentiate the unexpected NIC reset from the PCIe link fault.
19 . The apparatus of claim 18 , further configured to:
determine whether the hardware interrupt signals a NIC reset or a PCIe link fault based on the distinct interrupt signal;
execute a rapid cleanup of in-flight traffic through each PCIe link and each NIC corresponding to the fault condition, wherein executing the rapid cleanup comprises: identifying all input/output (IO) operations associated with a source of a fault notification corresponding to the fault condition; and issuing abort signals to terminate the identified IO operations to prevent data loss and system disruption;
disable data traffic paths through each PCIe link and each NIC corresponding to the fault condition, wherein if the fault notification corresponds to a NIC reset, all paths through a faulted NIC are disabled, and if the fault notification corresponds to a PCIe link fault, only paths associated with the PCIe link fault are disabled.
20 . The apparatus of claim 19 , further configured to:
maintain network operations through an operational NIC or PCIe link.