Cost reduced high reliability fault tolerant computer architecture
In part, in one aspect, the disclosure relates to a first computer system including a first processor and first memory, a first IO storage subsystem including a first switch configured for one or more first storage devices, a first IO non-storage subsystem including a first which configured for one or more first non-storage devices, a second compute system including a second processor and second memory, a second storage IO subsystem including a second switch configured for one or more second storage devices, a second IO non-storage subsystem including a second switch configured for one or more second non-storage devices and a midplane including a power connector, a processor side and an IO side, wherein the processing side includes connectors in electrical communication with the computer systems, the IO side includes connectors in electrical communication with the storage and non-storage subsystems.
1 . A fault tolerant system comprising:
a first computer system comprising a first processor, a first memory, and a first operating system configured to run on the first processor, the first operating system comprising a first platform driver;
a first IO storage subsystem comprising a first N-lane switch configured for one or more first storage devices;
a first IO non-storage subsystem comprising a first M-lane switch configured for one or more first non-storage devices, wherein M and N are whole numbers greater than or equal to 2;
a second computer system comprising a second processor, a second memory, and a second operating system configured to run on the second processor, the second operating system comprising a second platform driver;
a second storage IO subsystem comprising a second N-lane switch configured for one or more second storage devices;
a second IO non-storage subsystem comprising a second M-lane switch configured for one or more second non-storage devices;
wherein the first platform driver and the second platform driver are each configured to establish a communication channel between the first computer system and the second computer system,
wherein the first computer system and the second computer system are both in electrical communication with the first IO storage subsystem, the first IO non-storage subsystem, the second storage IO subsystem, and the second IO non-storage subsystem,
wherein one or more of the N-lane or M-lane switches of each computer system comprises a set of registers, wherein each computer system is configured to modify at least one register of the set of registers to communicate information to the other computer system; and
a midplane comprising a power connector, a processor side and an IO side, wherein the processor side comprises one or more connectors in electrical communication with the first computer system and the second computer system,
wherein the IO side comprises a plurality of connectors in electrical communication with the first IO storage subsystem, the first IO non-storage subsystem, the second storage IO subsystem, and the second IO non-storage subsystem.
2 . The system of claim 1 , wherein each N-lane switch and M-lane switch are unmanaged such that no processor in electrical communication with the midplane is managing any of the N-lane switches and the M-lane switches.
3 . The system of claim 1 , wherein each platform driver is a kernel mode driver.
4 . The system of claim 1 , wherein the information communicated by modifying registers includes messages for migrate operations, apply operations, and commit operations transmitted from the first computer system and acknowledgements of migrate operations, apply operations, and commit operations by the second computer system.
5 . The system of claim 1 , wherein each computer system switch comprises modified switch firmware.
6 . The system of claim 5 , wherein the modified switch firmware is configured to, in response to an indication that the first computer system is failing or about to fail, reprovision a group of devices in communication with the first computer system such that the group of devices becomes attached to and in communication with the second computer system.
7 . The system of claim 3 , where the first computer system comprises a switch, wherein the switch is connected to the midplane, wherein the switch comprises modified switch firmware, the modified switch firmware to generate a first synthetic device comprising a first set of registers.
8 . The system of claim 7 , wherein each platform driver is configured to write information to or read information from the first set of registers or a second set of registers of a second synthetic device, wherein the second synthetic device is connected to the second computer system.
9 . The system of claim 1 , wherein a first set of registers are written by the first computer system and read using a second set of registers that are paired with first set of registers to provide a communication channel between the first computer system and the second computer system.
10 . The system of claim 1 , wherein each M-lane switch and N-lane switch is a PCIe switch, wherein the first computer system comprises a first DMA engine and the second computer system comprises a second DMA engine.
11 . A method of exchanging information between an active compute node and a standby compute node of a fault tolerant system, the method comprising:
modifying a first bios of a first compute node and a second bios of a second compute node such that one or more parameters of the first compute node and the second compute node are synchronized;
establishing a DMA datapath between the first compute node and the second compute node;
establishing a host-to-host messaging datapath;
generating a migration request in response to the first compute node identifying a failure mode, wherein the first compute node is an active compute node;
querying a second compute node to assess standby compatibility, wherein the second compute node is a standby compute node;
establishing a PCI inventory on the first compute node;
establishing a plurality of operations that will be performed on the second compute node to bring the PCI inventory establish on the first compute node into service on the second compute node; and
transferring processor state information and memory data from a first computer system to a second computer system through the DMA datapath.
12 . The method of claim 11 further comprising:
confirming that a migration of data from the first compute node to the second compute node is acknowledged through the host-to-host messaging datapath; and
quiescing local devices connected to the first compute node and quiescing processor threads in the first compute node that are unrelated to the migration of data from the first compute node to the second compute node.
13 . The method of claim 12 further comprising:
applying processor state information and memory data to the second compute node; and
re-provisioning devices from the first compute node to the second compute node.
14 . The method of claim 13 further comprising:
verifying presence of re-provisioned devices in IO domain of second computer system; and
changing role of the second compute node to active.
15 . The method of claim 13 , wherein the re-provisioning step is performed at least in part using modified switch firmware of a second switch, wherein the second compute node comprises the second switch.
16 . A fault tolerant system comprising:
a first computer system comprising a first processor, a first bios, and a first memory;
a first IO subsystem comprising a first N-lane switch configured for one or more first devices, wherein the first N-lane switch comprises a set of registers;
a second computer system comprising a second processor, a second bios, and a second memory;
a second IO subsystem comprising a second N-lane switch configured for one or more second devices, wherein the second N-lane switch comprises a set of registers, wherein each computer system is configured to modify at least one register of the set of registers to communicate information to the other computer system; and
an IO side device, wherein the IO side device comprises a plurality of connectors in electrical communication with the first IO subsystem and the second IO subsystem, wherein the first bios and the second bios are modified such that one or more parameters of the first computer system and the second computer system are synchronized.