IP Library › Granted Patent US 12,565,240
Granted Patent B2
US 12,565,240 · App. 18/309,150 · Granted Mar 3, 2026

Method and system for graph neural network based pedestrian action prediction in autonomous driving systems

Inventors: Saber Malekmohammadi (Toronto, CA); Tiffany Yee Kay Yau (Markham, CA); Amir Rasouli (Maple, CA); Mohsen Rohani (Thornhill, CA); Jun Luo (Toronto, CA)
Assignee: HUAWEI TECHNOLOGIES CO., LTD.
B60W60/0027G06V10/7635G06V10/77G06V20/58B60W2554/4029B60W2554/4044B60W2554/4049
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,565,240
App. No.
18/309,150
Granted
Mar 3, 2026
Kind
B2
Abstract

The present disclosure relates to methods and systems for spatiotemporal graph modelling of road users in observed frames of an environment in which an autonomous vehicle operates (i.e. a traffic scene), clustering of the road users into categories, and providing the spatiotemporal graph to a trained graphical convolutional neural network (GNN) to predict a future pedestrian action. The future pedestrian action can be: one of the pedestrian will cross a road and the pedestrian will not cross the road. The spatiotemporal graph includes a better understanding of the observed frames (i.e. traffic scene).

Claims (47)

1 . A computer implemented method for predicting a pedestrian action, the method comprising:

receiving a temporal sequence of observed frames, each observed frame including spatial information for a target pedestrian and a plurality of road users;

for each observed frame in the sequence of observed frames:

encoding, based at least on the spatial information included in the observed frame, a set of target pedestrian features for the target pedestrian and a respective set of road user features for each of the plurality of road users;

generating, based at least on the spatial information included in the observed frame, a set of relative importance weights that includes, for each of the road users, a respective relative importance weight that indicates a relative importance of the road user to the target pedestrian, the respective relative importance weight for each road user being based both on a distance between the road user and the target pedestrian and a relative location importance of the road user to target pedestrian;

clustering, based on the spatial information included in multiple observed frames in the sequence including the observed frame, groups of road users from the plurality of road users into respective clusters based on behavioral similarities, wherein each of the respective clusters identifies a group of similar behaved road users;

predicting, based on the set of target pedestrian features encoded for each of a plurality of the observed frames, the respective sets of road user features encoded for each of the plurality of the observed frames, the set of relative importance weights generated for each of the plurality of the observed frames, and the respective clusters, a future action of the target pedestrian; and

automatically controlling an action of an autonomous vehicle based on the predicted future action of the target pedestrian.

2 . The method of claim 1 , wherein the relative location importance for each road user is based on a direction of movement of the road user relative to the target pedestrian.

3 . The method of claim 2 , wherein the relative location importance for each road user is given a greater importance if the road user is moving towards the target pedestrian than if the road user is moving away from the target pedestrian.

4 . The method of claim 2 , wherein the relative location importance for each road user is further based on a travel distance of the road user along a road relative to a position of the target pedestrian.

5 . The method of claim 2 , wherein relative location importance for each road user is based on a distance of the road user from a reference line that extends from the position of the target pedestrian and is perpendicular to a roadway direction of travel.

6 . The method of claim 1 , wherein, for each road user, the distance between the road user and the target pedestrian is a Euclidian distance.

7 . The method of claim 1 , wherein for each observed frame in the sequence of observed frames:

encoding the set of target pedestrian features for the target pedestrian and a respective set of road user features for each of the plurality of road users is based on the spatial information included in multiple observed frames in the sequence including the observed frame; and

generating the set of relative importance weights for each road user is based on the spatial information included in multiple observed frames in the sequence including the observed frame.

8 . The method of claim 1 , wherein a respective spatial graph is generated for each of the observed frames, wherein for each observed frame:

the respective spatial graph has a target pedestrian node representing the target pedestrian, and a plurality of road user nodes each representing a respective one of the plurality of road users, the respective spatial graph being defined by: (i) a feature matrix that includes the encoded target pedestrian features as features of the target pedestrian node, and includes the set of road user features encoded for the respective road users as features of the respective road user nodes; and (ii) an adjacency matrix that specifies: (a) respective weighted connecting edges between the target pedestrian node and each of the respective road user nodes corresponding to the set of relative importance weights generated for the observed frames; and (b) connecting edges between each of the road user nodes that are included in a respective cluster.

9 . The method of claim 8 , wherein predicting the future action of the target pedestrian is performed using a spatiotemporal convolutional graph neural network that receives the spatial graphs generated for the observed frames.

10 . The method of claim 1 , wherein the predicted pedestrian action is one of the pedestrian will cross in front of the autonomous vehicle or the pedestrian will not cross in front of the autonomous vehicle.

11 . The method of claim 1 , wherein for each observed frame in the sequence of observed frames: the set respective set of road user features encoded for each of the plurality of road users includes one or more of: a type of the road user; a location of the road user relative to the target pedestrian, a size of the road user, a velocity of the road user, and a direction of movement of the road user.

12 . A processing system comprising:

one or more processor systems;

one or more non-transitory memories storing instructions which when executed by the one or more processor systems cause the one or more processing systems to perform a method for predicting a pedestrian action comprising:

receiving a temporal sequence of observed frames, each observed frame including spatial information for a target pedestrian and a plurality of road users;

for each observed frame in the sequence of observed frames:

encoding, based at least on the spatial information included in the observed frame, a set of target pedestrian features for the target pedestrian and a respective set of road user features for each of the plurality of road users;

generating, based at least on the spatial information included in the observed frame, a set of relative importance weights that includes, for each of the road users, a respective relative importance weight that indicates a relative importance of the road user to the target pedestrian, the respective relative importance weight for each road user being based both on a distance between the road user and the target pedestrian and a relative location importance of the road user to target pedestrian;

clustering, based on the spatial information included in multiple observed frames in the sequence including the observed frame, groups of road users from the plurality of road users into respective clusters based on behavioral similarities, wherein each of the respective clusters identifies a group of similar behaved road users;

predicting, based on the set of target pedestrian features encoded for each of a plurality of the observed frames, the respective sets of road user features encoded for each of the plurality of the observed frames, the set of relative importance weights generated for each of the plurality of the observed frames, and the respective clusters, a future action of the target pedestrian; and

automatically controlling an action of an autonomous vehicle based on the predicted future action of the target pedestrian.

13 . The system of claim 12 , wherein the relative location importance for each road user is based on a direction of movement of the road user relative to the target pedestrian, and the relative location importance for each road user is given a greater importance if the road user is moving towards the target pedestrian than if the road user is moving away from the target pedestrian.

14 . The system of claim 13 , wherein the relative location importance for each road user is further based on a travel distance of the road user along a road relative to a position of the target pedestrian.

15 . The system of claim 14 , wherein relative location importance for each road user is based on a distance of the road user from a reference line that extends from the position of the target pedestrian and is perpendicular to a roadway direction of travel.

16 . The system of claim 12 , wherein for each observed frame in the sequence of observed frames, the method includes:

encoding the set of target pedestrian features for the target pedestrian and a respective set of road user features for each of the plurality of road users is based on the spatial information included in multiple observed frames in the sequence including the observed frame; and

generating the set of relative importance weights for each road user is based on the spatial information included in multiple observed frames in the sequence including the observed frame.

17 . The system of claim 12 , wherein a respective spatial graph is generated for each of the observed frames, wherein for each observed frame:

the respective spatial graph has a target pedestrian node representing the target pedestrian, and a plurality of road user nodes each representing a respective one of the plurality of road users, the respective spatial graph being defined by: (i) a feature matrix that includes the encoded target pedestrian features as features of the target pedestrian node, and includes the set of road user features encoded for the respective road users as features of the respective road user nodes; and (ii) an adjacency matrix that specifies: (a) respective weighted connecting edges between the target pedestrian node and each of the respective road user nodes corresponding to the set of relative importance weights generated for the observed frames; and (b) connecting edges between each of the road user nodes that are included in a respective cluster.

18 . A non-transitory computer-readable medium storing machine executable instructions which when executed by one or more processor systems cause the one or more processor systems to perform a method for predicting a pedestrian action comprising:

receiving a temporal sequence of observed frames, each observed frame including spatial information for a target pedestrian and a plurality of road users;

for each observed frame in the sequence of observed frames:

encoding, based at least on the spatial information included in the observed frame, a set of target pedestrian features for the target pedestrian and a respective set of road user features for each of the plurality of road users;

generating, based at least on the spatial information included in the observed frame, a set of relative importance weights that includes, for each of the road users, a respective relative importance weight that indicates a relative importance of the road user to the target pedestrian, the respective relative importance weight for each road user being based both on a distance between the road user and the target pedestrian and a relative location importance of the road user to target pedestrian;

clustering, based on the spatial information included in multiple observed frames in the sequence including the observed frame, groups of road users from the plurality of road users into respective clusters based on behavioral similarities, wherein each of the respective clusters identifies a group of similar behaved road users;

predicting, based on the set of target pedestrian features encoded for each of a plurality of the observed frames, the respective sets of road user features encoded for each of the plurality of the observed frames, the set of relative importance weights generated for each of the plurality of the observed frames, and the respective clusters, a future action of the target pedestrian; and

automatically controlling an action of an autonomous vehicle based on the predicted future action of the target pedestrian.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2023
From: MALEKMOHAMMADI, SABER; YAU, TIFFANY YEE KAY; RASOULI, AMIR; ROHANI, MOHSEN; LUO, JUN
To: HUAWEI TECHNOLOGIES CO., LTD.
Reel/Frame 064393/0771 →
Continuity (3)
Continuation PCTCA2021051539 · Nov 1, 2021
Provisional Application 63108336 · Oct 31, 2020
Related Publication 20230257003A1 · Aug 17, 2023
References Cited (32)
US 9495874B1 · Zhu et al. · 2016 [cited by applicant]
US 20100205132A1 · Taguchi · 2010 [cited by applicant]
US 20130054106A1 · Schmudderich et al. · 2013 [cited by applicant]
US 20170057497A1 · Laur · 2017 [cited by examiner]
US 20200023842A1 · Gutierrez · 2020 [cited by examiner]
US 20200218913A1 · Unnikrishnan · 2020 [cited by examiner]
US 20200223451A1 · Shashua · 2020 [cited by examiner]
US 20210103742A1 · Adeli-Mosabbeb · 2021 [cited by examiner]
US 20210201052A1 · Ranga · 2021 [cited by examiner]
US 20210295531A1 · Mangalam · 2021 [cited by examiner]
DE 102017004114A1 · 2018 [cited by applicant]
Dimitrievski et al, Behavioral Pedestrian Tracking Using a Camera and LiDAR Sensors on a Moving Vehicle, 2019, Sensors (2019), 19: 1-34. (Year: 2019). [cited by examiner]
Votz et al, Inferring Pedestrian Motions at Urban Crosswalks, 2019, IEEE Transactions on Intelligent Transportation Systems, 20(2):544-555. (Year: 2019). [cited by examiner]
Xue et al, Crossing-Road Pedestrian Trajectory Prediction via Encoder-Decoder LSTM, 2019, IEEE Intelligent Transportation Systems Conference, pp. 1-7. (Year: 2019). [cited by examiner]
T. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in ICLR, 2017. [cited by applicant]
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in International Conference on Knowledge Discovery and Data Mining, 1996. [cited by applicant]
Diehl, Frederik et al., “Graph Neural Networks for Modelling Traffic Participant Interaction” 2019 IEEE. [cited by applicant]
C. Yu, et.al., “Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction”, arXiv, Jul. 2020. [cited by applicant]
B. Liu, et.al., “Spatio-temporal Relationship Reasoning for Pedestrian Intent Prediction”, IEEE RA-L, Feb. 2020. [cited by applicant]
S. Haddad, et al., “Self-Growing Spatial Graph Networks for Pedestrian Trajectory Prediction”, IEEE WACV, Mar. 2020. [cited by applicant]
A. Mohammed, et al., “Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction”, CVPR, Jun. 2020. [cited by applicant]
J. Sun, et al., “Recursive Social Behavior Graph for Trajectory Prediction”, CVPR, Jun. 2020. [cited by applicant]
Hu, Yue et al., “Collaborative Motion Prediction via Neural Motion Message Passing” CVRP 2020 Paper. [cited by applicant]
Casas, Sergio et al., “IntentNet: Learning to Predict Intention from Raw Sensor Date” 2nd Conference on Robot Learning (CoRL 2018). [cited by applicant]
Kosaraju, Vineet et al., “Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks” NeurIPS 2019. [cited by applicant]
Mao, Wei et al., “Learning Trajectory Dependencies for Human Motion Prediction”. [cited by applicant]
Manh, Huynh et al., “Scene-LSTM: A model for Human Trajectory Prediction”. [cited by applicant]
S. Haddad, et.el., “Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model”, CVWW, Feb. 2019. [cited by applicant]
Yan, Sijie et al., “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition” AAAI-18. [cited by applicant]
Sun, Chen et al., “Relational Action Forecasting”. [cited by applicant]
H. Zhao, et.el., “TNT: Target-driveN Trajectory Prediction”, arXiv, Aug. 2020. [cited by applicant]
Zhang, Zhishuai et al., “STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Predication” CVRP 2020. [cited by applicant]