Contrastive training of object trajectory encoders and text encoders
Techniques are described herein for training contrastive models including object trajectory encoders and text encoders for evaluating, classifying, and/or predicting the movements and behaviors of dynamic objects in driving environments. A training system may receive sets of ground truth trajectory data describing movements of objects within driving environments, and associated text descriptions related to the trajectory data. The training system may jointly train the trajectory encoder and the text encoder, using contrastive loss, based on the related sets of trajectory data and text data. Once trained, the trajectory encoder and/or the text encoder may operate as pre-trained models for subsequently training and executing additional models with different output heads and/or various other downstream encoding tasks. In some examples, contrastive pre-trained trajectory encoders trained as described herein may be used for training and executing motion forecasting models within autonomous vehicles.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving driving scene data associated with a driving environment;
receiving an object trajectory of an object in the driving environment;
receiving a text description associated with the object trajectory of the object;
determining, using a trajectory encoder, and based at least in part on the driving scene data and the object trajectory, a first trajectory encoding;
determining, using a text encoder, and based at least in part on the text description associated with the object trajectory, a first text encoding; and
jointly training the trajectory encoder and the text encoder, wherein the jointly training comprises:
determining, based at least in part on a similarity between the first trajectory encoding and the first text encoding, a first loss associated with the trajectory encoder, and a second loss associated with the text encoder;
modifying the trajectory encoder, based at least in part on the first loss; and
modifying the text encoder, based at least in part on the second loss.
2 . The system of claim 1 , wherein the text description indicates at least one of:
a relationship between the object and a second object in the driving environment; or
a relationship between the object and a map data feature in the driving environment.
3 . The system of claim 1 , wherein the text encoder comprises:
a first set of transformer blocks associated with a large language model; and
a second set of transformer blocks associated with descriptions of object movements in the driving environment,
wherein jointly training the trajectory encoder and the text encoder comprises modifying the second set of transformer blocks.
4 . The system of claim 1 , wherein jointly training the trajectory encoder and the text encoder comprises:
training, during a first training stage, a trained trajectory encoder; and
wherein the operations further comprise:
training, during a second training stage after the first training stage, an object motion forecasting model including the trained trajectory encoder.
5 . The system of claim 4 , the operations further comprising:
transmitting the object motion forecasting model to a vehicle, wherein the vehicle is configured to be controlled based at least in part on the object motion forecasting model.