Monitoring processors operating in lockstep
An integrated circuit (IC) chip includes system circuitry having system memory, and a master processor and a checker processor configured to operate in lockstep; and monitoring circuitry comprising an internal lockstep monitor, a master tracer and a checker tracer. The internal lockstep monitor is configured to: observe states of internal signals of the master processor and the checker processor, compare corresponding observed states of the master processor and the checker processor, and if the corresponding observed states differ: trigger the master tracer to output stored master trace data recorded from the output of the master processor, and trigger the checker tracer to output stored checker trace data recorded from the output of the checker processor.
1. An integrated circuit (IC) chip comprising:
system circuitry comprising system memory, a master processor, and a checker processor configured to operate in lockstep; and
monitoring circuitry comprising an internal lockstep monitor, a master tracer, and a checker tracer, the internal lockstep monitor configured to:
observe states of internal signals of the master processor and the checker processor,
compare corresponding observed states of the master processor and the checker processor, and
when the corresponding observed states differ:
trigger the master tracer to output stored master trace data recorded from an output of the master processor, and
trigger the checker tracer to output stored checker trace data recorded from an output of the checker processor.
2. The IC chip of claim 1 , wherein the internal lockstep monitor is further configured to output the corresponding observed states when the corresponding observed states differ.
3. The IC chip of claim 1 , wherein the internal lockstep monitor comprises:
an internal master buffer configured to accumulate observed states of the master processor;
an internal checker buffer configured to accumulate observed states of the checker processor; and
a comparator configured to compare each observed state of the master processor from the internal master buffer with the corresponding observed state of the checker processor from the internal checker buffer.
4. The IC chip of claim 3 , wherein the internal lockstep monitor further comprises a timer configured to:
count an elapsed time between an observed state of the master processor being accumulated in the internal master buffer and an observed state of the checker processor being accumulated in the internal checker buffer; and
output an alert when the elapsed time exceeds a timeout threshold.
5. The IC chip of claim 1 , wherein the monitoring circuitry further comprises an output lockstep monitor configured to:
observe memory accesses requested by the master processor and the checker processor,
compare corresponding requested memory accesses of the master processor and the checker processor, and
when the corresponding requested memory accesses differ:
trigger the master tracer to output stored master trace data recorded from the output of the master processor,
trigger the checker tracer to output stored checker trace data recorded from the output of the checker processor, and
trigger the internal lockstep monitor to output observed states of the master processor and checker processor.
6. The IC chip of claim 5 , wherein the internal lockstep monitor is configured to, when the corresponding observed states differ, trigger the output lockstep monitor to output requested memory accesses of the master processor and the checker processor.
7. The IC chip of claim 1 , wherein the monitoring circuitry further comprises a lockstep access filter configured to:
receive the output of the master processor;
receive the output of the checker processor;
prevent requested memory accesses of the checker processor from propagating to the system memory; and
send messages to the checker processor from the system memory intended for the master processor.
8. The IC chip of claim 7 , wherein the internal lockstep monitor is configured to, when the corresponding observed states differ, trigger the lockstep access filter to output requested memory accesses of the master processor and the checker processor.
9. The IC chip of claim 1 , wherein the system circuitry further comprises a further checker processor configured to operate in lockstep with the master processor and the checker processor, and the internal lockstep monitor is configured to:
observe states of internal signals of the further checker processor,
compare corresponding observed states of the further checker processor, the master processor, and the checker processor, and
when any of the corresponding observed states differ:
trigger the master tracer to output stored master trace data recorded from the output of the master processor, and
trigger the checker tracer to output stored checker trace data recorded from the output of the checker processor.
10. The IC chip of claim 1 , wherein the internal lockstep monitor is configured to carry out the observing, the comparing, the triggering of the master tracer, and the triggering of the checker tracer at runtime of the master processor and the checker processor.
11. An integrated circuit (IC) chip comprising:
system circuitry comprising system memory, a master processor, and a checker processor configured to operate in lockstep; and
monitoring circuitry comprising an output lockstep monitor and analyzing circuitry, the output lockstep monitor configured to:
observe requested memory accesses of the master processor and the checker processor,
compare corresponding requested memory accesses of the master processor and the checker processor, and
trigger the analyzing circuitry to output recorded observed states of internal signals of the master processor and checker processor when the corresponding requested memory accesses differ.
12. The IC chip of claim 11 , wherein the monitoring circuitry further comprises a master tracer and a checker tracer, and
wherein, when the corresponding requested memory accesses differ, the output lockstep monitor is configured to:
trigger the master tracer to output stored master trace data recorded from the output of the master processor, and
trigger the checker tracer to output stored checker trace data recorded from the output of the checker processor.
13. The IC chip of claim 11 , wherein the analyzing circuitry comprises a master analyzer configured to observe states of internal signals of the master processor, and a checker analyzer configured to observe states of internal signals of the checker processor.
14. The IC chip of claim 11 , wherein the analyzing circuitry is an internal lockstep monitor configured to:
observe states of internal signals of the master processor and the checker processor,
compare corresponding observed states of the master processor and the checker processor, and
when the corresponding observed states differ:
trigger a master tracer to output stored master trace data recorded from an output of the master processor, and
trigger a checker tracer to output stored checker trace data recorded from an output of the checker processor.
15. The IC chip of claim 14 , wherein, when the corresponding observed states differ, the internal lockstep monitor is configured to trigger the output lockstep monitor to output requested memory accesses of the master processor and the checker processor.
16. The IC chip of claim 11 , wherein the output lockstep monitor is further configured to output the corresponding requested memory accesses when the corresponding requested memory accesses differ.
17. The IC chip of claim 11 , wherein the output lockstep monitor comprises:
an output master buffer configured to accumulate memory access requests of the master processor;
an output checker buffer configured to accumulate memory access requests of the checker processor; and
a comparator configured to compare each memory access request of the master processor from the output master buffer with the corresponding memory access request of the checker processor from the output checker buffer.
18. The IC chip of claim 17 , wherein the output lockstep monitor further comprises a timer configured to:
count an elapsed time between a memory access request of the master processor being accumulated in the output master buffer, and a memory access request of the checker processor being accumulated in the output checker buffer; and
output an alert when the elapsed time exceeds a timeout threshold.
19. The IC chip of claim 11 , wherein the monitoring circuitry further comprises a lockstep access filter configured to:
receive the output of the master processor;
receive the output of the checker processor;
prevent requested memory accesses of the checker processor from propagating to the system memory; and
send messages to the checker processor from the system memory intended for the master processor.
20. The IC chip of claim 19 , wherein the output lockstep monitor is configured to, when the corresponding requested memory accesses differ, trigger the lockstep access filter to output requested memory accesses of the master processor and the checker processor.
21. The IC chip of claim 11 , wherein the system circuitry further comprises a further checker processor configured to operate in lockstep with the master processor and the checker processor, and the output lockstep monitor is configured to:
observe requested memory accesses of the further checker processor, compare corresponding requested memory accesses of the further checker processor and master processor and checker processor, and
if any of the corresponding requested memory accesses differ, trigger the analyzing circuitry to output recorded observed states of internal signals of the master processor and checker processor and further checker processor.
22. The IC chip of claim 11 , wherein the output lockstep monitor is configured to carry out the observing, the comparing, the triggering at runtime of the master processor and the checker processor.