Cache coherency during resynchronization of self-correcting computer
View Patent ↗A fault-tolerant computer uses multiple commercial processors operating synchronously, i.e., in lock-step. In an exemplary embodiment, redundancy logic isolates the outputs of the processors from other computer components, so that the other components see only majority vote outputs of the processors. Processor resynchronization, initiated at predetermined time, milestones, and/or in response to processor faults, protects the computer from single event upsets. During resynchronization, processor state data is flushed and an instance of these data in accordance with processor majority vote is stored. Processor caches are flushed to update computer memory with more recent data stored in the caches. The caches are invalidated and disabled, and snooping is disabled. A controller is notified that snooping has been disabled. In response to the notification, the controller performs a hardware reset of the processors. The processors are loaded with the stored state data, and snooping and caches are enabled.
1. A method of operating a fault-tolerant computer system with a plurality of processors, the method comprising:
flushing out internal processor state data from processors of the plurality of processors;
determining an instance of the flushed out internal processor state data in accordance with processor majority vote;
storing the instance of the flushed out internal processor state data;
invalidating and disabling caches of the processors of the plurality of processors;
disabling snooping;
holding each processor of the plurality of processors in reset;
loading said each processor of the plurality of processors with the instance of the flushed out internal processor state data;
enabling snooping; and
enabling the caches of the processors of the plurality of processors;
wherein:
the step of disabling snooping is performed after the steps of invalidating and disabling caches;
the step of holding each processor of the plurality of processors in reset is performed after the step of disabling snooping;
the step of loading is performed after the step of holding each processor of the plurality of processors in reset; and
the step of enabling snooping is performed after the step of loading.
2. The method of claim 1 , further comprising:
synchronously operating the processors of the plurality of processors in parallel while determining processor majority vote of processor output signals.
3. A method of operating a fault-tolerant computer system with a plurality of processors, the method comprising:
flushing out internal processor state data from processors of the plurality of processors;
determining an instance of the flushed out internal processor state data in accordance with processor majority vote;
storing the instance of the flushed out internal processor state data;
invalidating and disabling caches of the processors of the plurality of processors;
disabling snooping;
holding each processor of the plurality of processors in reset;
loading said each processor of the plurality of processors with the instance of the flushed out internal processor state data;
enabling snooping;
enabling the caches of the processors of the plurality of processors;
first synchronously operating the processors of the plurality of processors in parallel while determining processor majority vote of processor output signals; and
second synchronously operating the processors of the plurality of processors in parallel while determining processor majority vote of processor output signals;
wherein:
the step of disabling snooping is performed after the steps of invalidating and disabling caches;
the step of holding each processor of the plurality of processors in reset is performed after the step of disabling snooping;
the step of loading is performed after the step of holding each processor of the plurality of processors in reset;
the step of enabling snooping is performed after the step of loading;
the step of first synchronously operating is performed before the step of flushing out internal processor state data; and
the step of second synchronously operating is performed after the step of loading.
4. The method of claim 3 , further comprising:
flushing out the caches of the processors of the plurality of processors between the step of flushing out internal processor state data from processors of the plurality of processors and the step of invalidating.
5. The method of claim 4 , wherein the step of flushing out internal processor slate data is performed at predetermined times.
6. The method of claim 4 , wherein the step of flushing out internal processor state data is performed at predetermined milestones.
7. The method of claim 4 , wherein the step of flushing out internal processor state data is performed at predetermined milestones of at least one software application executed by the computer system.
8. The method of claim 4 , wherein the step of flushing out internal processor state data is performed in response to a single event upset in at least one processor of the plurality of processors.
9. The method of claim 4 , wherein the step of flushing out internal processor state data is performed at predetermined intervals, further comprising:
shortening at least one of the intervals in response to an error in at least one processor of the plurality of processors.
10. The method of claim 4 , wherein the step of flushing out internal processor state data is performed in response to loss of synchronization of a first processor of the plurality of processors with respect to at least a second processor and a third processor of the plurality of processors.
11. The method of claim 4 , wherein the instance does not include data stored in the caches of the processors of the plurality of processors.
12. The method of claim 4 , wherein the instance includes all internal processor state data other than cache data stored in the caches of the processors of the plurality of processors.
13. A method of restoring synchronous operation of a plurality of processors in a computer system, the method comprising:
determining an instance of internal processor state data in accordance with processor majority vote of the plurality of processors;
storing the instance;
disabling caches of the processors of the plurality of processors;
after the step of disabling caches, disabling cache snooping;
after the step of disabling cache snooping, resetting each processor of the plurality of processors;
after the step of resetting, loading said each processor of the plurality of processors with the instance;
after the step of loading, enabling cache snooping;
after the step of enabling cache snooping, enabling the caches of the processors of the plurality of processors;
causing the processors of the plurality of processors to operate synchronously in parallel to execute application code.
14. The method of claim 13 , further comprising:
flushing out internal processor slate data before the step of resetting.
15. The method of claim 13 , further comprising:
flushing out internal processor state data before the step of resetting and after the step of disabling caches.
16. A fault-tolerant computer system comprising:
a plurality of processors configured to operate synchronously in parallel;
multi-module redundancy logic configured to receive outputs of each processor of the plurality of processors, and determine majority processor vote for each set of corresponding processor outputs; and
resynchronization state machine configured to
determine an instance of internal processor state data in accordance with processor majority vote of the plurality of processors,
store the instance,
disable caches of said each processor of the plurality of processors,
disable cache snooping after disabling caches,
reset said each processor of the plurality of processors after disabling cache snooping,
after resetting, load said each processor with the stored instance,
enable snooping after loading said each processor of the plurality of processors with the stored instance, and
enable the caches after enabling snooping,
whereby the resynchronization state machine restores synchronous operation of the plurality of processors and prevents snooping white the plurality of processors comes out of reset.
17. The computer system of claim 16 , further comprising:
a plurality of processor buses, wherein said each processor of the plurality of processors is coupled to the multi-module redundancy logic by a different processor bus of the plurality of processor buses.
18. The computer system of claim 17 , further comprising:
a processor correlation detector module configured to detect when at least one output of at least one processor of the plurality of processors differs from outputs of a plurality of corresponding outputs of the plurality of processors.
19. The computer system of claim 18 , further comprising:
a memory controller, wherein the multi-module redundancy logic provides the majority processor vote of at least a plurality of the outputs to the memory controller.
20. The computer system of claim 18 , further comprising:
a bus controller, wherein the mold-module redundancy logic provides the majority processor vote of at least a plurality of the outputs to the bus controller.
21. The computer system of claim 18 , further comprising:
a PCI bus controller, wherein the multi-module redundancy logic provides the majority processor vote of at least a plurality of the outputs to the PCI bus controller.
22. A fault-tolerant computer system comprising:
a plurality of processors configured to operate synchronously in parallel, wherein the processors may operate with enabled snooping, and wherein the processors may operate with disabled snooping;
means for receiving outputs of the processors, and for determining majority processor vote for each of the outputs; and
resynchronization means for synchronizing operation of the plurality of processors and preventing snooping by external devices during time periods when the processors are in reset and initialize coming out of reset.