System and method for a framework of robust and safe reinforcement learning application in real world autonomous vehicle application
In one aspect, a computer-implemented method useful for managing autonomous vehicle application operations with reinforcement learning (RL) methods, the method includes the step of providing an autonomous vehicle application of an autonomous vehicle, wherein the autonomous vehicle application manages a final action of a specified operation of the autonomous vehicle. The method includes the step of generating an RL model-agent for the specified operation. The RL model-agent learns by a maximizing rewards function related to the specified operation. The method includes the step of generating and managing a Safety Sanity Index (SSI) that monitors the safety performance of RL model. The method includes the step of obtaining an observed state of the autonomous vehicle, and generating an interruptible command based on the SSI and the observed state.
1. A computer-implemented method performed by an autonomous vehicle, the method comprising:
executing a reinforcement learning (RL) model-agent, the RL model-agent having been trained using a reinforcement learning method and a training dataset, wherein the RL model-agent is configured to determine one or more actions to be performed by the autonomous vehicle by maximizing a reward function related to a specified operation of the autonomous vehicle;
obtaining an observed state for the autonomous vehicle, wherein the observed state is based on captured sensor data, and wherein the observed state represents a current state of the autonomous vehicle and of one or more objects around the autonomous vehicle;
determining, by the RL model-agent and based on the observed state, a first action to be performed by the autonomous vehicle;
generating, by an Epistemic Uncertainty Check (EUC) module, an RL Model Confidence Factor (RLMCF) indicative of how well the RL model-agent can predict an action to be performed by the autonomous vehicle given the observed state, wherein generating the RLMCF comprises comparing a distribution of data in the observed state to a distribution of data in the training dataset;
communicating the RLMCF and the first action to an Action Integrity Module (AIM); and
determining, by the AIM and based on the RLMCF, a final action to be performed by the autonomous vehicle, wherein the final action is the first action or a different action that the AIM decides the autonomous vehicle should perform instead of the first action.
2. The computer-implemented method of claim 1 , further comprising:
determining a value of the reward function based on the first action;
passing the value of the reward function to the RL model-agent; and
determining, by the RL model-agent, a second action based on the value of the reward function, the second action to be performed by the autonomous vehicle after the final action.
3. The computer-implemented method of claim 2 , wherein the value of the reward function indicates how close the autonomous vehicle is to achieving a goal, wherein the value of the reward function is high when the first action brings the autonomous vehicle closer to achieving the goal and low or negative when the first action brings the autonomous vehicle farther from achieving the goal.
4. The computer-implemented method of claim 1 , wherein the RL model-agent comprises an artificial intelligence based software agent implemented in a computing system in the autonomous vehicle.
5. The computer-implemented method of claim 1 , wherein generating the RLMCF further comprises performing the following by the EUC module:
calculating a distance value indicating how much the distribution of data in the observed state varies from the distribution of data in the training dataset; and
determining that the distribution of data in the observed state varies from the distribution of data in the training dataset based on the distance value.
6. The computer-implemented method of claim 5 , wherein the distance value is a Mahalanobis distance.
7. The computer-implemented method of claim 1 , wherein the observed state comprises a current speed of the autonomous vehicle, a position of the autonomous vehicle, an acceleration of the autonomous vehicle, and a lane position of the autonomous vehicle.
8. The computer-implemented method of claim 1 , wherein comparing the distribution of data in the observed state to the distribution of data in the training dataset comprises:
comparing the sensor data on which the observed state is based to sensor data represented in the training dataset.
9. The computer-implemented method of claim 1 , further comprising:
generating a Safety Sanity Index (SSI) that is indicative of a safety performance of the RL model-agent;
generating an interruptible command based on the SSI and the observed state; and
communicating the interruptible command to the AIM, wherein determining the final action further comprises determining, based on the interruptible command, whether to override the first action with the different action.
10. A computer system comprising:
at least one processor; and
a memory containing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
executing a reinforcement learning (RL) model-agent, the RL model-agent having been trained using a reinforcement learning method and a training dataset, wherein the RL model-agent is configured to determine one or more actions to be performed by an autonomous vehicle by maximizing a reward function related to a specified operation of the autonomous vehicle;
obtaining an observed state for the autonomous vehicle, wherein the observed state is based on captured sensor data, and wherein the observed state represents a current state of the autonomous vehicle and of one or more objects around the autonomous vehicle;
determining, by the RL model-agent and based on the observed state, a first action to be performed by the autonomous vehicle;
generating, by an Epistemic Uncertainty Check (EUC) module, an RL Model Confidence Factor (RLMCF) indicative of how well the RL model-agent can predict an action to be performed by the autonomous vehicle given the observed state, wherein generating the RLMCF comprises comparing a distribution of data in the observed state to a distribution of data in the training dataset;
communicating the RLMCF and the first action to an Action Integrity Module (AIM); and
determining, by the AIM and based on the RLMCF, a final action to be performed by the autonomous vehicle, wherein the final action is the first action or a different action that the AIM decides the autonomous vehicle should perform instead of the first action.
11. The computer system of claim 10 , wherein the operations further comprise:
determining a value of the reward function based on the first action;
passing the value of the reward function to the RL model-agent; and
determining, by the RL model-agent, a second action based on the value of the reward function, the second action to be performed by the autonomous vehicle after the final action.
12. The computer system of claim 10 , wherein the generating of the RLMCF further comprises:
calculating a distance value indicating how much the distribution of data in the observed state varies from the distribution of data in the training dataset; and
determining that the distribution of data in the observed state varies from the distribution of data in the training dataset based on the distance value.
13. The computer system of claim 12 , wherein the distance value is a Mahalanobis distance.
14. The computer system of claim 10 , wherein comparing the distribution of data in the observed state to the distribution of data in the training dataset comprises:
comparing the sensor data on which the observed state is based to sensor data represented in the training dataset.
15. The computer system of claim 10 , wherein the operations further comprise:
generating a Safety Sanity Index (SSI) that is indicative of a safety performance of the RL model-agent;
generating an interruptible command based on the SSI and the observed state; and
communicating the interruptible command to the AIM, wherein determining the final action further comprises determining, based on the interruptible command, whether to override the first action with the different action.
16. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a computer system, cause the at least one processor to perform operations comprising:
executing a reinforcement learning (RL) model-agent, the RL model-agent having been trained using a reinforcement learning method and a training dataset, wherein the RL model-agent is configured to determine one or more actions to be performed by an autonomous vehicle by maximizing a reward function related to a specified operation of the autonomous vehicle;
obtaining an observed state for the autonomous vehicle, wherein the observed state is based on captured sensor data, and wherein the observed state represents a current state of the autonomous vehicle and of one or more objects around the autonomous vehicle;
determining, by the RL model-agent and based on the observed state, a first action to be performed by the autonomous vehicle;
generating, by an Epistemic Uncertainty Check (EUC) module, an RL Model Confidence Factor (RLMCF) indicative of how well the RL model-agent can predict an action to be performed by the autonomous vehicle given the observed state, wherein generating the RLMCF comprises comparing a distribution of data in the observed state to a distribution of data in the training dataset;
communicating the RLMCF and the first action to an Action Integrity Module (AIM); and
determining, by the AIM and based on the RLMCF, a final action to be performed by the autonomous vehicle, wherein the final action is the first action or a different action that the AIM decides the autonomous vehicle should perform instead of the first action.
17. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
generating the reward function based on the first action;
passing the reward function to the RL model-agent; and
determining, by the RL model-agent, a second action based on the reward function, the second action to be performed by the autonomous vehicle after the final action.
18. The non-transitory computer-readable medium of claim 16 , wherein the generating of the RLMCF further comprises:
calculating a distance value indicating how much the distribution of data in the observed state varies from the distribution of data in the training dataset; and
determining that the distribution of data in the observed state varies from the distribution of data in the training dataset based on the distance value.
19. The non-transitory computer-readable medium of claim 18 , wherein the distance value is a Mahalanobis distance.
20. The non-transitory computer-readable medium of claim, 16 wherein the operations further comprise:
generating a Safety Sanity Index (SSI) that is indicative of a safety performance of the RL model-agent;
generating an interruptible command based on the SSI and the observed state; and
communicating the interruptible command to the AIM, wherein determining the final action further comprises determining, based on the interruptible command, whether to override the first action with the different action.