Method and system for real-time diagnostics and fault monitoring in a robotic system
To determine a cause of a fault in a robotic system, a diagnostic service of the robotic system will receive primary signals from various processes running on the robotic system. The service will access a graph representation of functions as stored in memory, and the service will perform the functions on one or more of the primary signals to yield one or more derived signals. A sink of the robotic system will subscribing to a causal trace that includes a value for a specified one of the derived signals and an identification of each signal from which the specified derived signal was derived. During runtime, the sink will receive updates to the causal trace as the value for the specified derived signal changes. The diagnostic service will use the causal trace to identify a process that caused the fault.
1. A method of determining a cause of a fault in a robotic system, the method comprising:
by a diagnostic service of the robotic system:
receiving a plurality of primary signals from a plurality of processes running on the robotic system,
accessing a graph representation of a plurality of functions as stored in memory,
performing the functions on one or more of the primary signals to yield one or more derived signals, wherein at least a first derived signal of the one or more derived signals comprises a keyed signal with a plurality of first keys indicating which processes of the plurality of processes derived the primary signals used to yield the first derived signal,
performing aggregation operations to generate a second key based on the plurality of first keys, and
using an output of the aggregation operations to detect a fault in the robotic system;
by a sink of the robotic system:
subscribing to a causal trace, wherein the causal trace comprises a value for a specified derived signal of the derived signals and an identification of each signal from which the specified derived signal was derived,
during runtime, receiving updates to the causal trace as the value for the specified derived signal changes; and
by the diagnostic service, upon detection of the fault in the robotic system, using the causal trace to identify a process that caused the fault.
2. The method of claim 1 , wherein using the causal trace to identify a process that caused the fault comprises determining whether a processing latency in the causal trace exceeds a threshold time value.
3. The method of claim 1 , wherein using the causal trace to identify a process that caused the fault comprises determining whether an uncertainty in the causal trace exceeds a threshold value.
4. The method of claim 1 , wherein:
the causal trace further comprises a trigger; and
the sink only receives updates to the causal trace when the value for the specified derived signal changes and the trigger is active.
5. The method of claim 1 , wherein:
algorithms for the functions are stored by the diagnostic service prior to runtime as part of executable code; and
the method further comprises, by the diagnostic service, loading the configuration file to receive configuration variables to use with the functions at runtime.
6. The method of claim 1 , further comprising, by the diagnostic service,
causing the robotic system to take a safety-related action in response to the fault.
7. The method of claim 1 , wherein the robotic system comprises a vehicle, and the diagnostic service comprises an on-board vehicle diagnostic system.
8. The method of claim 6 , wherein:
the robotic system comprises a vehicle; and
the safety-related action comprises:
if the vehicle is not operating in an autonomous mode when the fault is detected, preventing the vehicle from entering the autonomous mode, or
if the vehicle is operating in the autonomous mode when the fault is detected, causing the vehicle to exit the autonomous mode.
9. A method determining a cause of a fault in a robotic system, the method comprising:
by a diagnostic service of the robotic system:
receiving a plurality of primary signals from a plurality of processes running on the robotic system,
performing one or more functions on one or more of the primary signals to yield one or more derived signals, wherein at least a first derived signal of the one or more derived signals comprises a keyed signal with a plurality of first keys indicating which processes of the plurality of processes derived the primary signals used to yield the first derived signal,
performing aggregation operations to generate a second key based on the plurality of first keys, and
using an output of the aggregation operations to detect a fault in the robotic system;
by a sink of the robotic system:
subscribing to a causal trace, wherein the causal trace comprises a value for a specified derived signal of the derived signals and an identification of each signal from which the specified signal was derived,
during runtime, receiving updates to the causal trace as the value for the specified derived signal changes; and
by the diagnostic service,
monitoring the processes running on the robotic system, and
upon detection of the fault in the robotic system:
using the causal trace to identify a process that caused the fault;
if the robotic system is not operating in an autonomous mode when the fault is detected, preventing the robotic system from entering the autonomous mode; and
if the robotic system is operating in the autonomous mode when the fault is detected, causing the robotic system to exit the autonomous mode.
10. The method of claim 9 , wherein using the causal trace to identify a process that caused the fault comprises one or more of the following:
determining whether a processing latency in the causal trace exceeds a threshold time value; or
determining whether an uncertainty in the causal trace exceeds a threshold value.
11. The method of claim 9 , wherein:
the causal trace further comprises a trigger; and
the sink only receives updates to the causal trace when the value for the specified derived signal changes and the trigger is active.
12. The method of claim 9 , wherein:
an algorithm for the function is stored by the diagnostic service prior to runtime as part of executable code; and
the method further comprises, by the diagnostic service, loading the configuration file to receive configuration variables to use with the function at runtime.
13. A system for determining a cause of a fault in a robotic device, the system comprising:
a diagnostic service comprising a processor and non-transitory memory containing programming instructions that are operable to cause the processor to:
receive a plurality of primary signals from a plurality of processes running on the robotic system,
access a graph representation of a plurality of functions as stored in memory, and
perform the functions on one or more of the primary signals to yield one or more derived signals, wherein at least a first derived signal of the one or more derived signals comprises a keyed signal with a plurality of first keys indicating which processes of the plurality of processes derived the primary signals used to yield the first derived signal,
perform aggregation operations to generate a second key based on the plurality of first keys, and
use an output of the aggregation operations to detect a fault in the robotic system; and
a sink that is operable to:
subscribe to a causal trace, wherein the causal trace comprises a value for a specified derived signal of the derived signals and an identification of each signal from which the specified signal was derived,
during runtime, receive updates to the causal trace as the value for the specified derived signal changes;
wherein the diagnostic service comprises additional programming instructions that are operable to cause the processor to, upon detection of the fault in the robotic device, use the causal trace to identify a process that caused the fault.
14. The system of claim 13 , wherein the programming instructions to use the causal trace to identify a process that caused the fault comprise instructions to determine whether a processing latency in the causal trace exceeds a threshold time value.
15. The system of claim 13 , wherein the programming instructions to use the causal trace to identify a process that caused the fault comprise instructions to determine whether an uncertainty in the causal trace exceeds a threshold value.
16. The system of claim 13 , wherein:
the causal trace further comprises a trigger; and
during operation, the sink will only receive updates to the causal trace when the value for the specified derived signal changes and the trigger is active.
17. The system of claim 13 , further comprising:
a memory that stores algorithms for the functions prior to runtime as part of executable code; and
additional programming instructions operable to cause the processor to load the configuration file to receive configuration variables to use with the functions at runtime.
18. The system of claim 13 , further comprising additional programming instructions that are configured to cause the processor to
cause the robotic device to take a safety-related action in response to the fault.
19. The system of claim 18 , wherein:
the robotic device comprises a vehicle; and
the safety-related action comprises:
if the vehicle is not operating in an autonomous mode when the fault is detected, preventing the vehicle from entering the autonomous mode, or
if the vehicle is operating in the autonomous mode when the fault is detected, causing the vehicle to exit the autonomous mode.
20. The system of claim 13 , wherein:
the robotic device comprises a vehicle; and
the diagnostic service comprises an on-board vehicle diagnostic system.