Vehicle operation with machine learning
A computer that includes a processor and a memory, the memory including instructions executable by the processor to operate a system based on predictions output from the machine learning system including predicted states, actions, rewards, and costs, wherein the machine learning system includes a first transformer and a second transformer and is trained based on bisimulation offline reinforcement learning, wherein the first transformer and the second transformer are based on a Markov decision process that includes the states, the actions, the rewards, and the costs. The bisimulation offline reinforcement learning can include inputting a first sequences of training states, actions, rewards, and costs to the first transformer and a second sequence of the training states, actions, rewards, and costs to the second transformer to determine bisimulation learning objectives based on latent variables output from the first transformer and the second transformer.
1 . A system, comprising: a computer that includes a processor and a memory, the memory including instructions executable by the processor to: operate the system based on output from a machine learning system including predicted states, actions, rewards, and costs, wherein the machine learning system includes a first transformer and a second transformer and is trained based on bisimulation offline reinforcement learning, wherein the first transformer and the second transformer are based on a Markov decision process that includes the states, the actions, the rewards, and the costs; and wherein the bisimulation offline reinforcement learning includes inputting a first sequence of training states, actions, rewards, and costs to the first transformer and a second sequence of the training states, actions, rewards, and costs to the second transformer to determine bisimulation learning objectives based on minimizing differences between latent variables including transition dynamics distributions, rewards, and costs output from the first transformer and latent variables including transition dynamics distributions, rewards, and costs output from the second transformer, respectively.
2 . The system of claim 1 , wherein the system includes a vehicle and operating the vehicle includes determining a vehicle trajectory based on the predicted states, actions, rewards, and costs output by the machine learning system.
3 . The system of claim 1 , wherein the Markov decision process is a constrained contextual Markov decision process that includes multiple sets of the states, the actions, the rewards, and the costs included in a time sequence.
4 . The system of claim 3 , wherein the Markov decision process includes the transition dynamics distributions and a discount factor.
5 . The system of claim 1 , wherein training the machine learning system based on the bisimulation offline reinforcement learning includes minimizing the bisimulation learning objectives based on the rewards, the costs, and the transition dynamics distributions included in the latent variables from the first transformer and the second transformer that includes a Lagrangian multiplier for the costs and a 2-Wasserstein distance for the transition dynamics distributions.
6 . The system of claim 1 , wherein the rewards are based on one or more of a vehicle longitudinal direction, a vehicle speed and a vehicle goal.
7 . The system of claim 1 , wherein the costs are based on one or more of not contacting objects including other vehicles, staying on a roadway, and maintaining an upper limit on vehicle speed.
8 . The system of claim 1 , wherein the states are based on a disjoint state space that includes a video image, a lidar image, and a bird's-eye view image.
9 . The system of claim 8 , wherein the bird's-eye view image is determined based on the video image and the lidar image.
10 . The system of claim 1 , wherein the first transformer and the second transformer transform the states, the actions, the rewards, and the costs to the predicted state based on encoding the states, the actions, the rewards, and the costs to a multi-dimensional vector, applying multi-head attention included in a decoder to the multi-dimensional vector to generate latent variables, and inputting the latent variables to an encoder that generates an output prediction.
11 . The system of claim 1 , wherein the sequences of the training states, the training actions, the training rewards, and the training costs used to train the first transformer and the second transformer are based on recorded real world data.
12 . A method, comprising:
operating a system based on output from a machine learning system including predicted states, actions, rewards, and costs, wherein the machine learning system includes a first transformer and a second transformer and is trained based on bisimulation offline reinforcement learning, wherein the first transformer and the second transformer are based on a Markov decision process that includes the states, the actions, the rewards, and the costs; and
wherein the bisimulation offline reinforcement learning includes inputting a first sequences of training states, actions, rewards, and costs to the first transformer and a second sequence of the training states, actions, rewards, and costs to the second transformer to determine bisimulation learning objectives based on minimizing differences between latent variables including transition dynamics distributions, rewards, and costs output from the first transformer and latent variables including transition dynamics distributions, rewards, and costs output from the second transformer, respectively.
13 . The method of claim 12 , wherein the system includes a vehicle and operating the vehicle includes determining a vehicle trajectory based on the predicted states, actions, rewards, and costs output by the machine learning system.
14 . The method of claim 12 , wherein the Markov decision process is a constrained contextual Markov decision process that includes multiple sets of the states, the actions, the rewards, and the costs included in a time sequence.
15 . The method of claim 14 , wherein the Markov decision process includes the transition dynamics distributions and a discount factor.
16 . The method of claim 12 , wherein training the machine learning system based on the bisimulation offline reinforcement learning includes minimizing the bisimulation learning objectives based on the rewards, the costs, and the transition dynamics distributions included in the latent variables from the first transformer and the second transformer that includes a Lagrangian multiplier for the costs and a 2-Wasserstein distance for the transition dynamics distributions.
17 . The method of claim 12 , wherein the rewards are based on one or more of a vehicle longitudinal direction, a vehicle speed and a vehicle goal.
18 . The method of claim 12 , wherein the costs are based on one or more of not contacting objects including other vehicles, staying on a roadway, and maintaining an upper limit on vehicle speed.
19 . The method of claim 12 , wherein the states are based on a disjoint state space that includes a video image, a lidar image, and a bird's-eye view image.
20 . The method of claim 19 , wherein the bird's-eye view image is determined based on the video image and the lidar image.