Fault isolation in an I/O fabric using downstream port containment
View Patent ↗A technique of non-disruptive fault isolation for an input/output (I/O) subsystem includes processing circuitry at a root complex interposed between a plurality of server platforms and a plurality of I/O switches receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled. In response to receipt of the notification, the processing circuitry notifies the plurality of attached server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter. The processing circuitry waits a containment interval, and thereafter, releases the containment to permit resetting of the I/O adapter.
1 . A computer-implemented method of non-disruptive fault isolation for an input/output (I/O) subsystem, the method comprising:
at a root complex interposed between a plurality of server platforms and a plurality of I/O switches, processing circuitry receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled;
in response to receipt of the notification, the processing circuitry notifying the plurality of server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter;
during an containment interval, a controlling server platform among the plurality of server platforms checking an I/O communication link coupling the I/O adapter and the I/O switch and collecting metadata regarding adapter operation from the I/O adapter for root-cause analysis while the I/O adapter is being subject to downstream port containment and the processing circuitry waiting a containment interval, and thereafter, releasing the containment to permit resetting of the I/O adapter, wherein the I/O adapter is a storage adapter; providing the notification to the plurality of server platforms comprises an MSI broadcast to all of the plurality of server platforms; and the containment interval terminates in response to a message from the controlling server platform.
2 . The method of claim 1 , further comprising:
the controlling server platform resetting the I/O adapter after elapsing of the containment interval.
3 . The method of claim 1 , further comprising:
a controlling server platform among the plurality of server platforms determining whether a port containment count for the I/O adapter satisfies a containment threshold; and
based on determining that the port containment count satisfies the containment threshold, the controlling server platform fencing the I/O adapter from receiving further I/O traffic and marking for physical replacement a field-replaceable unit including the I/O adapter.
4 . The method of claim 3 , wherein:
receiving notification of downstream port containment includes receiving a message signaled interrupt.
5 . The method of claim 3 , wherein:
releasing containment includes the processing circuitry signaling release of containment to both the I/O adapter and the plurality of server platforms.
6 . The method of claim 3 , wherein providing the notification to the plurality of server platforms includes broadcasting a message signaled interrupt to all of the plurality of server platforms.
7 . The method of claim 3 , wherein the containment interval terminates in response to a message from a controlling server platform among the plurality of server platforms.
8 . The method of claim 3 , wherein, in response to the notification, each of the plurality of server platforms marks the I/O adapter as subject to downstream port containment to refrain from attempting communication with the I/O adapter.
9 . The method of claim 3 , wherein enforcing the downstream port containment comprises: a digital signal processor of a downstream port enforcing port containment on an I/O link responsive to an uncorrectable error signaled as ERR_FATAL by the I/O adapter and reporting the port containment to the root complex in a message signaled interrupt.
10 . A computer program product, comprising: one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to perform computer operations, the computer operations including: at a root complex interposed between a plurality of server platforms and a plurality of I/O switches, processing circuitry receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled; in response to receipt of the notification, the processing circuitry notifying the plurality of server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter; during an containment interval, a controlling server platform among the plurality of server platforms checking an I/O communication link coupling the I/O adapter and the I/O switch and collecting metadata regarding adapter operation from the I/O adapter for root-cause analysis while the I/O adapter is being subject to downstream port containment; and the processing circuitry waiting a containment interval, and thereafter, releasing the containment to permit resetting of the I/O adapter, wherein the I/O adapter is a storage adapter; providing the notification to the plurality of server platforms comprises an MSI broadcast to all of the plurality of server platforms; and the containment interval terminates in response to a message from the controlling server platform.
11 . The computer program product of claim 10 , wherein the computer operations further comprise:
the controlling server platform resetting the I/O adapter after elapsing of the containment interval.
12 . The computer program product of claim 10 , wherein the computer operations further comprise:
a controlling server platform among the plurality of server platforms determining whether a port containment count for the I/O adapter satisfies a containment threshold; and
based on determining that the port containment count satisfies the containment threshold, the controlling server platform fencing the I/O adapter from receiving further I/O traffic and marking for physical replacement a field-replaceable unit including the I/O adapter.
13 . The computer program product of claim 10 , wherein:
receiving notification of downstream port containment includes receiving a message signaled interrupt.
14 . The computer program product of claim 10 , wherein:
releasing containment includes the processing circuitry signaling release of containment to both the I/O adapter and the plurality of server platforms.
15 . A data processing system, comprising: processing circuitry; and one or more computer-readable storage media communicatively coupled to the processing circuitry, wherein the one or more computer-readable storage media includes program instructions to perform operations including: at a root complex interposed between a plurality of server platforms and a plurality of I/O switches, processing circuitry receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled; in response to receipt of the notification, the processing circuitry notifying the plurality of server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter; during an containment interval, a controlling server platform among the plurality of server platforms checking an I/O communication link coupling the I/O adapter and the I/O switch and collecting metadata regarding adapter operation from the I/O adapter for root-cause analysis while the I/O adapter is being subject to downstream port containment; and the processing circuitry waiting a containment interval, and thereafter, releasing the containment to permit resetting of the I/O adapter, wherein the I/O adapter is a storage adapter; providing the notification to the plurality of server platforms comprises an MSI broadcast to all of the plurality of server platforms; and the containment interval terminates in response to a message from the controlling server platform.
16 . The data processing system of claim 15 , wherein the operations further comprise: the controlling server platform resetting the I/O adapter after elapsing of the containment interval.
17 . The data processing system of claim 15 , wherein the operations further comprise: a controlling server platform among the plurality of server platforms determining whether a port containment count for the I/O adapter satisfies a containment threshold; and based on determining that the port containment count satisfies the containment threshold, the controlling server platform fencing the I/O adapter from receiving further I/O traffic and marking for physical replacement a field-replaceable unit including the I/O adapter.
18 . The data processing system of claim 15 , wherein:
receiving notification of downstream port containment includes receiving a message signaled interrupt.
19 . The data processing system of claim 15 , wherein:
releasing containment includes the processing circuitry signaling release of containment to both the I/O adapter and the plurality of server platforms.