Pre-training of a reinforcement learning ground vehicle controller using monte carlo simulation
Techniques for utilizing a Monte Carlo model to perform pre-training of a ground vehicle controller. A sampled distribution of actions and corresponding states can be utilized to train a reinforcement learning controller policy, learn an action-value function, or select a set of control parameters with a predetermined loss.
1. A method comprising:
receiving a Monte Carlo generated stochastic control model of a set of ground vehicle drivers;
generating training data by sampling actions for given states of the Monte Carlo generated stochastic control model in a simulated ground vehicle environment and collecting traces of sequences of states, actions and rewards;
pre-training using the training data including the collected traces of sequences of states, actions, and rewards to learn a stochastic value function for expected discounted future returns; and
training a reinforcement learning controller using the stochastic value function.
2. The method according to claim 1 , wherein the Monte Carlo generated stochastic control model is parameterized by environment states.
3. The method according to claim 1 , wherein the generated training data is parametric or non-parametric.
4. The method according to claim 1 , wherein the pre-training comprises supervised learning.
5. The method according to claim 1 , wherein generation of the stochastic value function includes a regression function to determine an value function Q that minimizes the difference between the determined discounted expected future return R and the predicted discounted expected return {circumflex over (R)}.
6. The method according to claim 1 , wherein the reinforcement learning controller is trained using the value function.
7. The method according to claim 1 , wherein Monte Carlo generation of the stochastic control model comprises:
generating, utilizing a Monte Carlo simulation, a set of possible driving scenarios, wherein the scenarios include environment, vehicle slates, controls and related performance metric data;
selected a predetermined subset of driving scenarios from the set of possible driving scenarios; and
fitting a stochastic control model to the selected predetermined subset of driving scenarios.
8. The method according to claim 7 , wherein:
the performance metric data comprises fuel efficiency; and
the predetermined subset of driving scenarios is selected based on a predetermined fuel efficiency range.
9. The method according to claim 1 , wherein Monte Carlo generation of the stochastic control model comprises:
generating, utilizing a Monte Carlo simulation, a set of possible driving scenarios, wherein the scenarios include environment, vehicle states, controls and related performance metric data;
selected a predetermined subset of driving scenarios from the set of possible driving scenarios; and
inferring a stochastic control model from the selected predetermined subset of driving scenarios.
10. The method according to claim 9 , wherein:
the performance metric data comprises fuel efficiency; and
the predetermined subset of driving scenarios is selected based on a predetermined fuel efficiency range.
11. A method comprising:
receiving a stochastic Monte Carlo inferred model of a set of ground vehicle drivers;
generating training data by sampling actions for given states of the stochastic Monte Carlo model in a simulated ground vehicle environment and collecting traces of sequences of states and probabilities of actions;
pre-training using the training data including the collected traces of sequences of states and probability of action to generate a stochastic policy; and
training a reinforcement learning controller using the stochastic policy.
12. The method according to claim 11 , wherein the Monte Carlo model is parameterized by environment states.
13. The method according to claim 11 , wherein the generated training data is parametric or non-parametric.
14. The method according to claim 11 , wherein the pre-training comprises supervised learning.
15. The method according to claim 11 , wherein generation of the stochastic policy includes minimizing a distribution dissimilarity metric.
16. The method according to claim 15 , wherein the distribution dissimilarity metric comprises a cross entropy measurement.
17. The method according to claim 11 , wherein the reinforcement learning controller is trained using the stochastic policy.
18. The method according to claim 11 , wherein Monte Carlo generation of the stochastic control model comprises:
generating, utilizing a Monte Carlo simulation, a set of possible driving scenarios, wherein the scenarios include environment, vehicle states, controls and related performance metric data;
selected a predetermined subset of driving scenarios from the set of possible driving scenarios; and
fitting a stochastic control model to the selected predetermined subset of driving scenarios.
19. The method according to claim 18 , wherein:
the performance metric data comprises fuel efficiency; and
the predetermined subset of driving scenarios is selected based on a predetermined fuel efficiency range.
20. The method according to claim 11 , wherein Monte Carlo generation of the stochastic control model comprises:
generating, utilizing a Monte Carlo simulation, a set of possible driving scenarios, wherein the scenarios include environment, vehicle states, controls and related performance metric data;
selected a predetermined subset of driving scenarios from the set of possible driving scenarios; and
inferring a stochastic control model from the selected predetermined subset of driving scenarios.
21. The method according to claim 20 , wherein:
the performance metric data comprises fuel efficiency; and
the predetermined subset of driving scenarios is selected based on a predetermined fuel efficiency range.