Vision-language-planning (VLP) models with agent-wise learning for autonomous driving
Methods and systems for training an autonomous driving, agent-centric vison-language planning (VLP) machine learning model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating BEV features including spatiotemporal information linked to the vehicle and the recognized agents. Executing the VLP model begins by first extracting agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment. Agent-wise text features are extracted from natural language text prompts. A contrastive learning model derives similarities between the agent-wise BEV features and the agent-wise text features. The BEV features are refined based on the similarities, and the refined BEV features are used in various models to generate a modified predicted trajectory of the vehicle.
1 . A method of training an autonomous driving system utilizing a vision-language planning (VLP) machine learning model with agent-wise learning, the method comprising:
receiving image data generated from a camera mounted to a vehicle, wherein the image data includes agents in an environment outside the vehicle;
via image processing, detecting the agents in the environment based on the image data;
generating a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes BEV features including spatiotemporal information associated with the vehicle and the detected agents;
inputting data from the BEV into a perception model, a prediction model, and a planning model of an end-to-end autonomous driving system to generate a predicted trajectory of the vehicle; and
executing an agent-centric vision-language planning (VLP) machine learning model to:
extract agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment,
generate natural language text prompts associated with the agents in the environment,
extract agent-wise text features from the natural language text prompts, wherein the agent-wise text features are associate with the respective agents in the environment,
execute a contrastive learning model to derive similarities between the agent-wise BEV features and the agent-wise text features, wherein the contrastive learning model includes (i) a text encoder configured to output a text-based vector representing the agent-wise text features associated with the natural language text prompts, and (ii) an image encoder configured to output an image-based vector representing agent-wise BEV features associated with the agents in the BEV, and wherein execution of the contrastive learning model evaluates similarities between the text-based vector and the image-based vector, and
refine the BEV features used for the perception model, the prediction model, and the planning model based on the similarities to generate a modified predicted trajectory of the vehicle.
2 . The method of claim 1 , further comprising:
determining a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle; and
repeat the steps of claim 1 until convergence to minimize the loss.
3 . The method of claim 1 , wherein the natural language text prompts are generated using a template and ground truth information associated with the environment existent in training data.
4 . The method of claim 1 , wherein an output of the contrastive learning is used as a training loss for training the VLP machine learning model.
5 . The method of claim 1 , wherein the contrastive learning model is further configured to execute a dot product to evaluate the similarities between the text-based vector and the image-based vector.
6 . The method of claim 1 , wherein the contrastive learning model is further configured to push apart dissimilarities between the agent-wise BEV features and the agent-wise text features.
7 . The method of claim 1 , further comprising:
repeating of the executing of the agent-centric VLP machine learning model until convergence; and
outputting a trained agent-centric VLP machine learning model based on the convergence.
8 . A system utilizing a vision-language planning (VLP) machine learning model, the system comprising:
a camera mounted to a vehicle and configured to generate image data associated with agents in an environment outside the vehicle;
a processor; and
memory including instructions that, when executed by the processor, cause the processor to:
process the image data to detect agents in the environment,
generate a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes BEV features including spatiotemporal information associated with the vehicle and the detected agents,
input data from the BEV into a perception model, a prediction model, and a planning model of an end-to-end autonomous driving system to generate a predicted trajectory of the vehicle, and
execute an agent-centric vision-language planning (VLP) machine learning model to:
(i) extract agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment,
(ii) generate natural language text prompts associated with the agents in the environment,
(iii) extract agent-wise text features from the natural language text prompts, wherein the agent-wise text features are associate with the respective agents in the environment,
(iv) execute a contrastive learning model to derive similarities between the agent-wise BEV features and the agent-wise text features, wherein the contrastive learning model includes (a) a text encoder configured to output a text-based vector representing the agent-wise text features associated with the natural language text prompts, and (b) an image encoder configured to output an image-based vector representing agent-wise BEV features associated with the agents in the BEV, wherein execution of the contrastive learning model evaluates similarities between the text-based vector and the image-based vector, and
(v) refine the BEV features used for the perception model, the prediction model, and the planning model based on the similarities to generate a modified predicted trajectory of the vehicle.
9 . The system of claim 8 , wherein the memory includes further instructions that, when executed by the processor, cause the processor to:
determine a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle; and
execute the VLP model until convergence to minimize loss.
10 . The system of claim 8 , wherein the natural language text prompts are generated using a template and ground truth information associated with the environment existent in training data.
11 . The system of claim 8 , wherein an output of the contrastive learning is used as a training loss for training the VLP machine learning model.
12 . The system of claim 8 , wherein the contrastive learning model is further configured to execute a dot product to evaluate the similarities between the text-based vector and the image-based vector.
13 . The system of claim 8 , wherein the contrastive learning model is further configured to push apart dissimilarities between the agent-wise BEV features and the agent-wise text feature.
14 . The system of claim 8 , wherein the memory includes further instructions that, when executed by the processor, cause the processor to:
repeat the executing of the agent-centric VLP machine learning model until convergence; and
output a trained agent-centric VLP machine learning model based on the convergence.
15 . The system of claim 8 , wherein the agents in the environment include at least one of a pedestrian, another vehicle, or a cyclist.
16 . An apparatus for training at least one machine learning model, the apparatus comprising:
a processor; and
memory including instructions that, when executed by the processor, cause the processor to:
process image data generated from a camera mounted to a vehicle in order to detect agents in the environment,
generate a bird eye view (BEV) of the environment based on the image data, wherein the BEV includes BEV features including spatiotemporal information associated with the vehicle and the detected agents,
input data from the BEV into a perception model, a prediction model, and a planning model of an end-to-end autonomous driving system to generate a predicted trajectory of the vehicle, and
execute an agent-centric vision-language planning (VLP) machine learning model to:
(i) extract agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment,
(ii) generate natural language text prompts associated with the agents in the environment,
(iii) extract agent-wise text features from the natural language text prompts, wherein the agent-wise text features are associate with the respective agents in the environment,
(iv) execute a contrastive learning model to derive similarities between the agent-wise BEV features and the agent-wise text features, wherein the contrastive learning model includes (a) a text encoder configured to output a text-based vector representing the agent-wise text features associated with the natural language text prompts, and (b) an image encoder configured to output an image-based vector representing agent-wise BEV features associated with the agents in the BEV, wherein execution of the contrastive learning model evaluates similarities between the text-based vector and the image-based vector, and
(v) refine the BEV features used for the perception model, the prediction model, and the planning model based on the similarities to generate a modified predicted trajectory of the vehicle.
17 . The apparatus of claim 16 , wherein the memory includes further instructions that, when executed by the processor, cause the processor to:
determine a loss between the predicted trajectory of the vehicle and a ground truth trajectory of the vehicle; and
execute the VLP model until convergence to minimize loss.