Systems and methods for maintaining lock step operation
View Patent ↗A system is provided for rapidly synchronizing two or more processing elements in a fault-tolerant computing system. Embodiments of this system allow for the rapid synchronization of two processing elements through partial copies of the contents of memory associate with each processing element.
1. A microsync method of returning a fault tolerant system to lock step operation, the fault tolerant system having two processing subsystems, the method comprising the steps of:
detecting an error, the error corresponding to an out of lock event in the fault tolerant system;
halting processing in the two processing subsystems;
waiting for a delay period, the delay period set to allow for a declaration of a hardware error;
setting processing subsystem priority after the delay period such that one processing subsystem is primary and one processing subsystem is secondary;
copying processor state information for the primary subsystem to the secondary subsystem; and
returning the primary subsystem and the secondary subsystem to normal operation within a re-sync period.
2. The method of claim 1 wherein the re-sync period ranges from about 10 milliseconds to about 200 milliseconds.
3. The method of claim 1 wherein the delay period is less than about 90 milliseconds.
4. The method of claim 1 wherein the step of setting processing subsystem priority is performed if a memory for each of the processing subsystems is identical.
5. The method of claim 1 further comprising the steps of
if an error is detected, disabling one or more steps in the microsync method until a memory verification step is successfully completed.
6. A method of regulating lock step operation in a fault tolerant system, wherein the fault tolerant system comprises a first processor and a second processor, the first and second processors each having an internal state and memory, the method comprising the steps of:
halting further processing by the first and second processors in response to an error such that contents of the memories associated with the first processor and the second processor remain identical;
adjusting the internal state of the second processor such that it corresponds identically with the internal state of the first processor;
copying I/O transactions from memory associated with the first processor to memory associated with the second processor to ensure identical processing queues after re-sync; and
re-syncing the first and second processors and restoring lock step operation within a re-sync period.
7. The method of claim 6 wherein the re-syncing comprises completing a partial copy of the contents of memory associated with the first processor to memory associated with the second processor.
8. A fault tolerant system adapted to maintain lock step operation having a system clock, the system comprising:
a first processing subsystem having a first internal state that changes over time, and a first memory element;
a second processing subsystem having a second internal state that changes over time, and a second memory element; and
a fault tolerant logic component in electrical communication with the first and second subsystems, the fault tolerant logic component adapted to:
a) halt the processing subsystems and prevent further internal state changes;
b) replicate processing transactions between the processing subsystems;
c) regulate lockstep operation of the two processing subsystems; and
d) re-sync the first and second subsystems in lockstep in accordance with a particular cycle of the system clock within a re-sync period.
9. The fault tolerant system of claim 8 wherein the re-sync period ranges from about 10 milliseconds to about 200 milliseconds.
10. The fault tolerant system of claim 8 wherein the fault tolerant logic component is selected from the group consisting of a VLSI circuit, an ASIC, a motherboard chipset, software, firmware, and combinations thereof.