IP Library › Granted Patent US 12,528,516
Granted Patent B2
US 12,528,516 · App. 18/335,920 · Granted Jan 20, 2026

Structured multi-agent interactive trajectory forecasting

Inventors: Wenjie Luo (Santa Clara, CA); Cheolho Park (Palo Alto, CA); Dragomir Anguelov (San Francisco, CA); Benjamin Sapp (Marina del Rey, CA)
Assignee: Waymo LLC
B60W60/00272B60W50/0097B60W2554/4026B60W2554/4029B60W2554/4046
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,528,516
App. No.
18/335,920
Granted
Jan 20, 2026
Kind
B2
Abstract

Methods, systems, and apparatus for generating trajectory predictions for one or more agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents. The one or more computers process the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines multiple possible trajectories for the agent after the current time point and a respective likelihood score for each of the multiple possible future trajectories. The one or more computers can generate graph data based on the respective marginal trajectory predictions, and the one or more computers can process the graph data using a graph neural network to generate a joint trajectory prediction output for the multiple agents in the scene.

Claims (53)

1 . A method performed by one or more computers, the method comprising:

obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents;

processing the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines (i) a plurality of possible future trajectories for the agent after the current time point and (ii) a respective likelihood score for each of the plurality of possible future trajectories;

generating, based on the respective marginal trajectory predictions for the plurality of agents, graph data representing a graph of the scene that comprises a respective node for each of the plurality of agents and edges that each connect a respective pair of nodes, the generating comprising:

determining whether to connect nodes representing a pair of agents of the plurality of agents with an edge based on respective marginal trajectory predictions for the pair of agents;

processing the graph data using a graph neural network to generate a joint trajectory prediction output for the plurality of agents in the scene; and

providing at least one of (i) the joint trajectory prediction output or (ii) data derived from the joint trajectory prediction output to an on-board system of an autonomous vehicle for use in controlling the autonomous vehicle.

2 . The method of claim 1 , wherein:

the scene context data comprises data generated from data captured by one or more sensors of an autonomous vehicle, and

the agents are agents in a vicinity of the autonomous vehicle in the environment.

3 . The method of claim 2 , wherein the joint trajectory prediction output is generated on-board the autonomous vehicle.

4 . The method of claim 1 , wherein:

the context data comprises data generated from data that simulates data that would be captured by one or more sensors of an autonomous vehicle in the real-world environment, and

the target agent is a simulated agent in a vicinity of the simulated autonomous vehicle in the computer simulation.

5 . The method of claim 4 , further comprising:

providing (i) the joint trajectory prediction output, (ii) data derived from the joint trajectory prediction output, or (iii) both for use in controlling the simulated autonomous vehicle in the computer simulation.

6 . The method of claim 1 , wherein determining whether to connect nodes representing the pair of agents with an edge based on the respective marginal trajectory predictions for the pair of agents comprises:

identifying a respective highest-scoring possible future trajectory for each of the agents in the pair according to the likelihood scores in the respective marginal trajectory prediction for the agent; and

determining whether to connect nodes representing the pair of agents with an edge based on a similarity between the respective highest-scoring possible future trajectories.

7 . The method of claim 1 , wherein the graph also includes a node representing a conditioning agent and the future behavior prediction for the agent is fixed to a conditioning future trajectory for the conditioning agent.

8 . The method of claim 7 , wherein the conditioning agent is an autonomous vehicle.

9 . The method of claim 1 , wherein the graph also includes a node representing a target agent and the graph is a star-graph that connects the node representing the target agent to all other nodes in the graph.

10 . The method of claim 9 , wherein the target agent is an autonomous vehicle.

11 . The method of claim 1 , wherein generating, based on the respective marginal trajectory predictions, graph data representing a graph of the scene that comprises a respective node for each of the plurality of agents and edges that each connect a respective pair of nodes comprises:

generating, based on the respective marginal trajectory predictions, respective node features for each of the plurality of nodes and respective edge features for each of the plurality of edges.

12 . The method of claim 11 , wherein generating, based on the respective marginal trajectory predictions, respective node features for each of the plurality of nodes and respective edge features for each of the plurality of edges comprises:

for each of the plurality of nodes, determining a unary potential feature for the node from the likelihood scores in the marginal trajectory prediction for the agent represented by the node.

13 . The method of claim 12 , wherein generating, based on the respective marginal trajectory predictions, respective node features for each of the plurality of nodes and respective edge features for each of the plurality of edges comprises:

for each of the plurality of edges, determining a pairwise potential feature for the edge from the marginal trajectory predictions for the pair of agents represented by the nodes connected by the edge.

14 . The method of claim 13 , wherein determining a pairwise potential feature for the edge from the marginal trajectory predictions for the agents represented by the nodes connected by the edge comprises:

generating transformed trajectories for the first agent in the pair by transforming the predicted future trajectories in the marginal behavior prediction for the first agent into the a coordinate system centered at the second agent in the pair;

generating transformed trajectories for the second agent in the pair by transforming the predicted future trajectories in the marginal behavior prediction for the second agent into a coordinate system centered at the first agent in the pair; and

generating the pairwise potential feature from the transformed trajectories for the first agent and the transformed trajectories for the second agent.

15 . The method of claim 14 , wherein generating the pairwise potential feature from the transformed trajectories for the first agent and the transformed trajectories for the second agent comprises:

processing each pair of transformed trajectories that includes a first agent trajectory and a second agent trajectory using a potential prediction neural network to generate a respective predicted potential score for the pair.

16 . The method of claim 1 , wherein the joint trajectory prediction output identifies a most likely future trajectory for each agent given the marginal trajectory predictions for the plurality of agents.

17 . The method of claim 16 , wherein the joint trajectory prediction output approximates a joint probability distribution over future trajectories for the plurality of agents.

18 . The method of claim 17 , wherein the joint probability distribution is a joint probability distribution over the possible future trajectories in the marginal trajectory predictions for the plurality of agents.

19 . A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents;

processing the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines (i) a plurality of possible future trajectories for the agent after the current time point and (ii) a respective likelihood score for each of the plurality of possible future trajectories;

generating, based on the respective marginal trajectory predictions for the plurality of agents, graph data representing a graph of the scene that comprises a respective node for each of the plurality of agents and edges that each connect a respective pair of nodes, the generating comprising:

determining whether to connect nodes representing the pair of agents of the plurality of agents with an edge based on the respective marginal trajectory predictions for the pair of agents;

processing the graph data using a graph neural network to generate a joint trajectory prediction output for the plurality of agents in the scene; and

providing at least one of (i) the joint trajectory prediction output or (ii) data derived from the joint trajectory prediction output to an on-board system of the autonomous vehicle for use in controlling the autonomous vehicle.

20 . One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents;

processing the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines (i) a plurality of possible future trajectories for the agent after the current time point and (ii) a respective likelihood score for each of the plurality of possible future trajectories;

generating, based on the respective marginal trajectory predictions for the plurality of agents, graph data representing a graph of the scene that comprises a respective node for each of the plurality of agents and edges that each connect a respective pair of nodes, the generating comprising:

determining whether to connect nodes representing the pair of agents of the plurality of agents with an edge based on the respective marginal trajectory predictions for the pair of agents;

processing the graph data using a graph neural network to generate a joint trajectory prediction output for the plurality of agents in the scene; and

providing (i) the joint trajectory prediction output or (ii) data derived from the joint trajectory prediction output to an on-board system of the autonomous vehicle for use in controlling the autonomous vehicle.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 28, 2023
From: LUO, WENJIE; PARK, CHEOLHO; ANGUELOV, DRAGOMIR; SAPP, BENJAMIN
To: WAYMO LLC
Reel/Frame 065687/0606 →
Continuity (2)
Provisional Application 63352618 · Jun 15, 2022
Related Publication 20230406361A1 · Dec 21, 2023
References Cited (48)
US 11498587B1 · Mitlin · 2022 [cited by examiner]
US 11975726B1 · Gu · 2024 [cited by examiner]
US 12204823B1 · Capell · 2025 [cited by examiner]
US 12208819B1 · Funke · 2025 [cited by examiner]
US 20220355825A1 · Deo · 2022 [cited by examiner]
Alahi et al., “Social LSTM: Human trajectory prediction in crowded spaces,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 27-30, 2016, pp. 961-971. [cited by applicant]
Bansal et al., “ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst,” CoRR, Dec. 7, 2018, arXiv: 1812.03079v1, 20 pages. [cited by applicant]
Bhat et al., “Trajformer: Trajectory prediction with local self-attentive contexts for autonomous driving,” CoRR, submitted on Nov. 30, 2020, arXiv:2011.14910v1, 6 pages. [cited by applicant]
Caesar et al., “nuScenes: A multimodal dataset for autonomous driving,” CoRR, Mar. 26, 2019, arXiv:1903.11027v1, 10 pages. [cited by applicant]
Casas et al., “IntentNet: Learning to predict intention from raw sensor data,” Proceedings of the 2nd Conference on Robot Learning (PMLR), Oct. 23, 2018, 87:947-956. [cited by applicant]
Casas et al., “Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data,” CoRR, submitted on Oct. 18, 2019, arXiv:1910.08233v1, 11 pages. [cited by applicant]
Chang et al., “Argoverse: 3d tracking and forecasting with rich maps,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 16-20, 2019, pp. 8748-8757. [cited by applicant]
Cui et al., “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” 2019 International Conference on Robotics and Automation (ICRA), May 20-24, 2019, pp. 2090-2096. [cited by applicant]
Ettinger et al., “Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset,” CoRR, submitted on Apr. 20, 2021, arXiv:2104.10133v1, 15 pages. [cited by applicant]
Gao et al., “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 13-19, 2020, pp. 11525-1153. [cited by applicant]
Gilles et al., “GOHOME: Graph-oriented heatmap output for future motion estimation,” CoRR, submitted on Sep. 4, 2021, arXiv:2109.01827v1, 14 pages. [cited by applicant]
Girgis et al., “Autobots: Latent variable sequential set transformers,” CoRR, Jun. 16, 2021, arXiv:2104.00563v2, 21 pages. [cited by applicant]
Gu et al., “DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 11-17, 2021, pp. 15303-15312. [cited by applicant]
Gu et al., “DenseTNT: Waymo Open Dataset Motion Prediction Challenge 1st Place Solution,” CoRR, submitted on Sep. 26, 2021, 5 pages. [cited by applicant]
Ho et al., “Axial attention in multidimensional transformers,” CoRR, submitted on Dec. 20, 2019, arXiv: 1912.12180v1, 11 pages. [cited by applicant]
Khandelwal et al., “What-if motion prediction for autonomous driving,” CoRR, submitted on Aug. 24, 2020, arXiv:2008.10587v1, 16 pages. [cited by applicant]
Lee et al., “DESIRE: Distant future prediction in dynamic scenes with interacting agents,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21-26, 2017, pp. 336-345. [cited by applicant]
Li et al., “RAIN: Reinforced hybrid attention inference network for motion forecasting,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 11-17, 2021, pp. 16096-16106. [cited by applicant]
Liang et al., “Learning lane graph representations for motion forecasting,” Proceeding of the Computer Vision—ECCV 2020: 16th European Conference, Aug. 23-28, 2020, Aug. 2020, pp. 541-556. [cited by applicant]
Liang et al., “The garden of forking paths: Towards multi-future trajectory prediction,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 13-19, 2020, pp. 10508-10518. [cited by applicant]
Liu et al., “Multimodal motion prediction with stacked transformers,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 19-25, 2021, pp. 7577-7586. [cited by applicant]
Luo et al., “Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), J… [cited by applicant]
Marchetti et al., “MANTRA: Memory augmented networks for multiple trajectory prediction,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 13-19, 2020, pp. 7143-7152. [cited by applicant]
Mercat et al., “Multi-head attention for multi-modal joint vehicle motion forecasting,” 2020 IEEE International Conference on Robotics and Automation (ICRA), May 31, 2020, pp. 9638-9644. [cited by applicant]
Mo et al., “Heterogeneous Edge-Enhanced Graph Attention Network for Multi-Agent Trajectory Prediction,” CoRR, submitted on Jun. 14, 2021, arXiv:2106.07161v1, 12 pages. [cited by applicant]
Ngiam et al., “Scene transformer: A unified multi-task model for behavior prediction and planning,” CoRR, submitted on Jun. 15, 2021, arXiv:2106.08417v1, 21 pages. [cited by applicant]
Phan-Minh et al., “CoverNet: Multimodal behavior prediction using trajectory sets,” CoRR, submitted on Nov. 23, 2019, arXiv:1911.10298v1, 12 pages. [cited by applicant]
Rhinehart et al. “PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27-Nov. 2, 2019, pp. 2821-2830. [cited by applicant]
Rhinehart et al., “R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting,” Proceedings of the European Conference on Computer Vision (ECCV), Sep. 8-14, 2018, pp. 794-811. [cited by applicant]
Salzmann et al., “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data,” CoRR, submitted on Apr. 5, 2020, arXiv:2001.03093v2, 22 pages. [cited by applicant]
Sapp et al., “Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,” Proceedings of the Conference on Robot Learning (PMLR), Nov. 2019, 100:86-99. [cited by applicant]
Song et al., “PiP: Planning-informed trajectory prediction for autonomous driving,” Computer Vision—ECCV 2020: 16th European Conference, Aug. 23-28, 2020, pp. 598-614. [cited by applicant]
Sun et al., “M2I: From factored marginal trajectory prediction to interactive prediction,” CoRR, submitted on Mar. 28, 2022, arXiv:2202.11884v2, 15 pages. [cited by applicant]
Tang et al., “Multiple futures prediction,” Proceedings of the 33rd International Conference on Neural Information Processing Systems, Dec. 2019, pp. 15424-15434. [cited by applicant]
Tolstaya et al., “Identifying driver interactions via conditional behavior prediction,” CoRR, submitted on Jun. 1, 2021, arXiv:2104.09959v2, 7 pages. [cited by applicant]
Van Den Oord et al., “Neural discrete representation learning,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp. 6309-6318. [cited by applicant]
Varadarajan et al., “Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction,” CoRR, submitted on Dec. 22, 2021, arXiv:2111.14973v3, 22 pages. [cited by applicant]
Yuan et al., “Diverse trajectory forecasting with determinantal point processes,” Proceeding of International Conference on Learning Representations, Apr. 2020, 15 pages. [cited by applicant]
Zeng et al., “DSDNet: Deep Structured Self-Driving Network,” Proceeding of Computer Vision—ECCV 2020: 16th European Conference, Aug. 23-28, 2020, pp. 156-172. [cited by applicant]
Zeng et al., “End-to-end interpretable neural motion planner,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 16-20, 2019, pp. 8660-8669. [cited by applicant]
Zeng et al., “LaneRCNN: Distributed representations for graphcentric motion forecasting,” CoRR, submitted on Jan. 17, 2021, arXiv:2101.06653v1, 14 pages. [cited by applicant]
Zhan et al., “Interaction Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps,” CoRR, Sep. 30, 2019, arXiv:1910.03088v1, 13 pages. [cited by applicant]
Zhao et al., “TNT: Target-driveN trajectory prediction,” CoRR, submitted on Aug. 19, 2020, arXiv:2008.08294v1, 12 pages. [cited by applicant]