Checkpoint-based tracing for monitoring a robotic system

1. A method of analyzing program execution in a robotic device, the method comprising:

by an operating system comprising processor and that is communicatively connected to a plurality of sensors and a plurality of components of a robotic device:

receiving sensor data from the plurality of sensors of the robotic device;

for each sensor:

generating a data log comprising an identifier of a first checkpoint associated with that sensor and a first timestamp,

performing an execution flow comprising one or more processes on the sensor data from that sensor, and

updating the data log to include an identifier and time stamp for one or more additional checkpoints during the execution flow,

fusing results from each execution flow to yield fused data;

using the fused data as an input for a decision process;

causing a component of the robotic device to take an action in response to an output of the decision process; and

recording the action, an action timestamp and the data logs for each sensor in a memory.

2. The method of claim 1 , further comprising, by the operating system:

determining whether, for any sensor, a difference between the action timestamp and the first timestamp for that sensor exceeds a threshold; and

for any sensor whose difference exceeds the threshold, using the data log for that sensor to trace the execution flow for that sensor and identify a cause of latency in that execution flow.

3. The method of claim 1 , wherein the action comprises causing the robotic device to operate in a safe mode.

4. The method of claim 1 , wherein updating the data log for each sensor to include an identifier and timestamp for one or more additional checkpoints during the execution flow comprises doing so only for checkpoints that are associated with processes that may affect latency in that execution flow.

5. The method of claim 1 , wherein performing the execution flow and updating the data log for each sensor comprises, at each of a plurality of checkpoints in the execution flow, updating the data log and passing the data log along to a next checkpoint in the execution flow without retaining a copy of the data log.

6. The method of claim 1 further comprising, by the operating system prior to recording the action timestamp, determining an estimated action timestamp for a corresponding action.

7. The method of claim 6 further comprising, by the operating system:

determining whether the corresponding action occurred at a time of the estimated action timestamp;

upon determining that the corresponding action did not occur at the estimated action timestamp, causing the robotic device to take a corrective action.

8. The method of claim 1 , wherein fusing the results from each execution flow to yield the fused sensor data comprises storing one or more identifying elements for two or more of the checkpoints in the execution flow.

9. The method of claim 1 , wherein fusing the results from each execution flow to yield the fused sensor data comprises, at one of the additional checkpoints, storing an identifier of the first checkpoint, along with a timestamp for one or more data elements that flowed from the first checkpoint.

10. The method of claim 1 , wherein fusing the results from each execution flow to yield the fused sensor data further comprises storing timestamps for data that flowed from checkpoints that are ancestor checkpoints to the first checkpoint in the execution flow.

11. An automated device monitoring system comprising:

a processor;

a communication link between the processor and a plurality of sensors of a robotic device; and

a memory device containing programming instructions that are configured to cause the processor to:

receive, via the communication link, sensor data from the sensors of the robotic device,

for each sensor:

generate a data log comprising an identifier of a first checkpoint associated with that sensor and a first timestamp;

perform an execution flow comprising one or more processes on the sensor data from that sensor; and

update the data log to include an identifier and time stamp for one or more additional checkpoints during the execution flow,

fuse results from each execution flow to yield fused data,

use the fused data as an input for a decision process,

cause a component of the robotic device to take an action in response to an output of the decision process, and

record the action, an action timestamp and the data logs for each sensor in a data store.

12. The system of claim 11 , further comprising additional programming instructions that are configured to cause the processor to:

determine whether, for any sensor, a difference between the action timestamp and the first timestamp for that sensor exceeds a threshold; and

for any sensor whose difference exceeds the threshold, use the data log for that sensor to trace the execution flow for that sensor and identify a cause of latency in that execution flow.

13. The system of claim 11 , wherein the programming instructions to cause a component of the robotic device to take an action comprises instructions to cause the robotic device to operate in a safe mode.

14. The system of claim 11 , wherein the programming instructions to update the data log for each sensor to include an identifier and timestamp for one or more additional checkpoints during the execution flow comprise instructions to do so only for checkpoints that are associated with processes that may affect latency in that execution flow.

15. The system of claim 11 , wherein the programming instructions to perform the execution flow and update the data log for each sensor comprise instructions to, at each of a plurality of checkpoints in the execution flow, update the data log and pass the data log along to a next checkpoint in the execution flow without retaining a copy of the data log.

16. The system of claim 11 further comprising additional programming instructions that are configured to cause the processor to, prior to recording the action timestamp, determine an estimated action timestamp for a corresponding action.

17. The system of claim 16 further comprising additional programming instructions that are configured to cause the processor to:

determine whether the corresponding action occurred at a time of the estimated action timestamp; and

upon determining that the corresponding action did not occur at the estimated action timestamp, cause the robotic device to take a corrective action.

18. The system of claim 11 , wherein the programming instructions to fuse the results from each execution flow to yield the fused sensor data comprise instructions to store one or more identifying elements for two or more of the checkpoints in the execution flow.

19. The system of claim 11 , wherein the programming instructions to fuse the results from each execution flow to yield the fused sensor data comprise instructions to, at one of the additional checkpoints, store an identifier of the first checkpoint, along with a timestamp for one or more data elements that flowed from the first checkpoint.

20. The system of claim 11 , wherein the programming instructions to fuse the results from each execution flow to yield the fused sensor data further comprise instructions to store timestamps for data that flowed from checkpoints that are ancestor checkpoints to the first checkpoint in the execution flow.