Concurrent repair of the PCIe switch units in a tightly-coupled, multi-switch, multi-adapter, multi-host distributed system
Techniques are disclosed to perform an operation to facilitate concurrent repair of PCIe switch units in processing environments such as a tightly coupled, multi-switch, multi-adapter, multi-host distributed system. The operation, for an identified switch unit to be repaired, reconfigures all switch unit hardware in the switch fabric by removing all upstream to downstream connections utilizing the identified switch unit. Connections to hosts via the upstream ports are also removed by the operation. Once the switch unit is powered back on, the operation reconfigures all switch unit hardware in the switch fabric by adding all upstream to downstream connections utilizing the identified switch unit. The operation further restores connections to hosts via the upstream ports.
1. A computer-implemented method to repair switch units in a distributed switch comprising a plurality of switch units, each switch unit of the plurality having at least one port for establishing connections according to a predefined interface, the method comprising:
responsive to receiving an indication to vary off a first switch unit of the plurality of switch units:
identifying a host connected to the first switch unit;
transmitting a first removal indication to the host to remove a connection between the host and the first switch unit; and
upon determining that: (i) the host is connected to the first switch unit through a downstream port of the first switch unit and (ii) the host has not acknowledged the first removal indication within a predefined amount of time, transmitting a second removal indication by operation of one or more computer processors, wherein the second removal indication emulates that the first switch unit is physically removed from the distributed switch, wherein the first switch unit is not physically removed from the distributed switch; and
responsive to receiving an indication to vary on the first switch unit:
transmitting an add indication to the host to establish a connection between the host and the first switch unit.
2. The method of claim 1 , wherein the predefined interface is a Peripheral Component Interconnect (PCI) Express interface, wherein the second removal indication is a PCIe standard hot plug controller (SHPC) manual retention latch (MRL) open command.
3. The method of claim 2 , wherein the host is connected to a second switch unit, wherein an upstream port of the second switch unit is connected to the downstream port of the first switch unit, wherein the first removal indication is an SHPC attention command, wherein transmitting the add indication comprises:
transmitting, to the host, an SHPC MRL closed command emulating that the first switch element is connected to the switch fabric; and
transmitting, to the host, an SHPC attention command indicating that the host create a connection to the first switch element.
4. The method of claim 1 , wherein the first switch unit is connected to the host through an upstream port of the first switch unit, wherein the first removal indication is an out-of-band hot plug remove command, wherein the add indication is an out-of-band hot plug add command, wherein the method further comprises:
upon determining that the host has not acknowledged the first removal indication within a predefined amount of time, aborting the operation to vary off the first switch unit.
5. The method of claim 4 , wherein the operation to vary off the first switch unit is not aborted if a force vary off indication is provided with the indication to vary off the first switch unit, wherein the force vary off indication varies off the first switch unit even if the host does not acknowledge the first removal indication.
6. The method of claim 1 , wherein the first removal indication and the second removal indication reconfigure the plurality of switch units to remove existing connections between a respective upstream port of each of the plurality of switch units and the downstream port of the first switch unit.
7. The method of claim 6 , wherein the add indication operates to restore the removed connections between the respective upstream port of each of the plurality of switch units and the downstream port of the first switch unit.