Reinforcement learning based control of imitative policies for autonomous driving
A method for controlling an ego agent includes periodically receiving policy information comprising a spatial environment observation and a current state of the ego agent. The method also includes selecting, for each received policy information, a low-level policy from a number of low-level policies. The low-level policy may be selected based on a high-level policy. The method further includes controlling an action of the ego agent based on the selected low-level policy.
1 . A method for selecting a driving mode by a semi-autonomous vehicle, comprising:
periodically receiving policy information comprising a respective position and a respective velocity of each agent within a given distance from the semi-autonomous vehicle and a current position and a current velocity of the semi-autonomous vehicle within an environment;
selecting, in accordance with a high-level policy, a driving mode of a group of driving modes that maximizes a reward from a reward function that is based on a conditional observation probability of the policy information, the reward function accounting for resource use, a time to a destination, and collision avoidance, the group of driving modes including at least an aggressive driving mode and a defensive driving mode, the conditional observation probability modeling uncertainty due to sensor noise, the conditional observation probability defining a probability of observing the policy information given an unobserved behavior of one or more agents in the environment;
selecting an action from a plurality of actions associated with the driving mode, each driving mode of the group of driving modes associated with a respective low-level policy from a plurality of low-level policies trained via imitation learning, the imitation learning comprising supervised training of a policy model to directly map observed states to expert-labeled actions from mode-specific demonstrations provided by a group of human operators, each one of the group of human operators manually operating a training vehicle in accordance with one or more driving modes of the group of driving modes, the action including a throttle and a steering command that adjusts one or both of a current velocity or a current driving direction of the semi-autonomous vehicle, the action being selected in accordance with the reward function; and
overriding instructions from the human operator of the vehicle, such that the vehicle performs the action.
2 . The method of claim 1 , further comprising training, via reinforcement learning, a high-level policy model to learn the high-level policy based on each of the plurality of low-level policies.
3 . The method of claim 2 , in which training via the reinforcement learning comprises learning to maximize the reward function based on an action determined from a partially observable Markov decision process.
4 . An apparatus for selecting a driving mode for a semi-autonomous vehicle, comprising:
at least one processor;
at least one memory coupled with the at least one processor; and
instructions stored in the memory and operable, when executed by the at least one processor, to cause the apparatus to:
periodically receive policy information comprising a respective position and a respective velocity of each agent within a given distance from the semi-autonomous vehicle and a current position and a current velocity of the semi-autonomous vehicle within an environment;
select, in accordance with a high-level policy, a driving mode of a group of driving modes that maximizes a reward from a reward function that is based on a conditional observation probability of the policy information, the reward function accounting for resource use, a time to a destination, and collision avoidance, the group of driving modes including at least an aggressive driving mode and a defensive driving mode, the conditional observation probability modeling uncertainty due to sensor noise, the conditional observation probability defining a probability of observing the policy information given an unobserved behavior of one or more agents in the environment;
select an action from a plurality of actions associated with the driving mode, each driving mode of the group of driving modes associated with a respective low-level policy from a plurality of low-level policies trained via imitation learning, the imitation learning comprising supervised training of a policy model to directly map observed states to expert-labeled actions from mode-specific demonstrations provided by a group of human operators, each one of the group of human operators manually operating a training vehicle in accordance with one or more driving modes of the group of driving modes, the action including a throttle and a steering command that adjusts one or both of a current velocity or a current driving direction of the semi-autonomous vehicle, the action being selected in accordance with the reward function; and
override instructions from the human operator of the vehicle, such that the vehicle performs the action.
5 . The apparatus of claim 4 , in which the instructions further cause the apparatus to train, via reinforcement learning, a high-level policy model to learn the high-level policy based on each of the plurality of low-level policies.
6 . The apparatus of claim 5 , in which the instructions further cause the apparatus to train via the reinforcement learning comprises learning to maximize the reward function based on an action determined from a partially observable Markov decision process.
7 . A non-transitory computer-readable medium having program code recorded thereon for selecting a driving mode for a vehicle, the program code executed by at least one processor and comprising:
program code to periodically receive policy information comprising a respective position and a respective velocity of each agent within a given distance from the semi-autonomous vehicle and a current position and a current velocity of the semi-autonomous vehicle within an environment;
program code to select, in accordance with a high-level policy, a driving mode of a group of driving modes that maximizes a reward from a reward function that is based on a conditional observation probability of the policy information, the reward function accounting for resource use, a time to a destination, and collision avoidance, the group of driving modes including at least an aggressive driving mode and a defensive driving mode, the conditional observation probability modeling uncertainty due to sensor noise, the conditional observation probability defining a probability of observing the policy information given an unobserved behavior of one or more agents in the environment;
program code to select an action from a plurality of actions associated with the driving mode, each driving mode of the group of driving modes associated with a respective low-level policy from a plurality of low-level policies trained via imitation learning, the imitation learning comprising supervised training of a policy model to directly map observed states to expert-labeled actions from mode-specific demonstrations provided by a group of human operators, each one of the group of human operators manually operating a training vehicle in accordance with one or more driving modes of the group of driving modes, the action including a throttle and a steering command that adjusts one or both of a current velocity or a current driving direction of the semi-autonomous vehicle, the action being selected in accordance with the reward function; and
program code to override instructions from the human operator of the vehicle, such that the vehicle performs the action.
8 . The non-transitory computer-readable medium of claim 7 , in which the program code further comprises program code to train, via reinforcement learning, a high-level policy model to learn the high-level policy based on each of the plurality of low-level policies.
9 . The non-transitory computer-readable medium of claim 8 , in which the program code to train via the reinforcement learning comprises program code to learn to maximize the reward function based on an action determined from a partially observable Markov decision process.
10 . The method of claim 1 , in which the reward functions maximizes a return based on minimizing a travel time to a destination and maximizing safety of the vehicle.
11 . The apparatus of claim 4 , in which the reward functions maximizes a return based on minimizing a travel time to a destination and maximizing safety of the vehicle.
12 . The method of claim 1 , in which the high-level policy is periodically selected in response to a trigger associated with the policy information.
13 . The apparatus of claim 4 , in which the high-level policy is periodically selected in response to a trigger associated with the policy information.
14 . The non-transitory computer-readable medium of claim 7 , in which the high-level policy is periodically selected in response to a trigger associated with the policy information.