IP Library › Granted Patent US 12,711,676
Granted Patent B2
US 12,711,676 · App. 18/466,870 · Granted Aug 18, 2026

Methods and systems for generating graph representations of a scene

Inventors: Sriram Vaikundam (Singapore, SG); Michael Colin Hoy (Singapore, SG); Paththini Gedara Chaminda Namal Senarathne (Singapore, SG); Rahul Singh (Singapore, SG)
Assignee: AUMOVIO Germany GmbH
G06T11/26G06V10/426G06V10/82G06V2201/07
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,711,676
App. No.
18/466,870
Filed
Sep 14, 2023
Granted
Aug 18, 2026
Kind
B2
Art Unit
2617
USPC
345/440
Abstract

A computer-implemented system and method of generating a graph representation of a scene comprising receiving sensor data representative of a perceived scene captured with a sensor; defining a plurality of nodes based on the received sensor data; creating a densely connected graph by connecting each node to a number of nearest neighbour nodes; predicting, for each pair of connected nodes of the densely connected graph, at least a node probability, wherein the node probability represents a probability that the pair of connected nodes represent the same object; and generating a graph representation of the perceived scene based at least on the densely connected graph, and the predicted node probability. The method may also be applied to two or more observations of a scene.

Claims (12)

1 . A computer-implemented method of generating a graph representation of a scene based on a single observation of the scene, said graph representation comprising a plurality of nodes and edges, wherein a node represents an object within the scene, and wherein an edge connects pairs of nodes of the plurality of nodes and represents a spatial relationship between objects within the scene, the method comprising:

receiving sensor data representative of a perceived scene captured with a sensor;

defining a plurality of nodes based on the received sensor data;

creating a densely connected graph by connecting each node to a number of nearest neighbour nodes;

predicting, for each pair of connected nodes of the densely connected graph, at least a node probability, wherein the node probability represents a probability that the pair of connected nodes represent the same object; and

generating a graph representation of the perceived scene based at least on the densely connected graph, and the predicted node probability.

2 . The computer-implemented method of claim 1 , wherein defining a plurality of nodes based on the captured perceived scene comprises:

identifying one or more objects within the perceived scene based on the received sensor data;

determining whether each identified object is an object of interest, wherein an object of interest is preferably a static object; and

defining a plurality of nodes, wherein each node corresponds to an object of interest within the perceived scene.

3 . The computer-implemented method of claim 1 , further comprising merging the pair of connected nodes when the node probability is above a first threshold.

4 . The computer-implemented method of claim 1 , further comprising removing an edge from between the pair of nodes when an edge probability of the edge between the pair of nodes is below a second threshold.

Assignments (2)
CHANGE OF NAME Recorded May 13, 2026
From: CONTINENTAL AUTOMOTIVE TECHNOLOGIES GMBH
To: AUMOVIO GERMANY GMBH
Reel/Frame 075608/0635 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 14, 2023
From: VAIKUNDAM, SRIRAM; HOY, MICHAEL COLIN; SENARATHNE, PATHTHINI GEDARA CHAMINDA NAMAL; SINGH, RAHUL
To: CONTINENTAL AUTOMOTIVE TECHNOLOGIES GMBH
Reel/Frame 064898/0807 →
Priority Claims (1)
EP 20195493 · Sep 14, 2022 · regional
Continuity (1)
Related Publication 20240087189A1 · Mar 14, 2024
References Cited (31)
US 20020158880A1 · Williams · 2002 [cited by examiner]
US 20210150771A1 · Huang · 2021 [cited by applicant]
US 20210192254A1 · Rejeb Sfar · 2021 [cited by examiner]
US 20210295093A1 · Pan · 2021 [cited by examiner]
US 20220067233A1 · Blackwell, II · 2022 [cited by examiner]
CN 103248170A · 2013 [cited by applicant]
CN 110598770A · 2019 [cited by applicant]
European Search Report dated Mar. 15, 2023 for European Patent Application No. 22195493.6 to which this application claims priority. [cited by applicant]
Yang Jianwei et al., “Graph R-CNN for Scene Graph Generation”, Oct. 6, 2018 (Oct. 6, 2018), Oct. 6, 2018, pp. 690-706, XP047635262, [retrieved on Oct. 6, 2018]. [cited by applicant]
Shih-Yuan Yu et al., “Scene-Graph Augmented Data-Driven Risk Assessment of Autonomous Vehicle Decisions”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Aug. 31, 2020 (Aug. … [cited by applicant]
Arnav V Malawade et al: “Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Nov. 11, 2021 (No… [cited by applicant]
Monica Riccardo et al: “Multi-label Point Cloud Annotation by Selection of Sparse Control Points”, 2017 International Conference on 3D Vision (3DV), IEEE, Oct. 10, 2017 (Oct. 10, 2017), pp. 301-308, XP033353200, DOI: 10… [cited by applicant]
Charles R Qi et al: “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Dec. 2, 2016 (Dec. 2, 2016… [cited by applicant]
Nawaz Waqas et al., “Intra graph clustering using collaborative similarity measure”, Distributed and Parallel Databases, Kluwer, NL, vol. 33, No. 4, Jan. 20, 2015 (Jan. 20, 2015), pp. 583-603, XP035530950, ISSN: 0926-87… [cited by applicant]
Thomas N Kipf et al., “Semi-Supervised Classification with Graph Convolutional Networks”, Conference ICLR 2017, Feb. 22, 2017 (Feb. 22, 2017), XP055457092. [cited by applicant]
Danfei Xu et al., “Scene Graph Generation by Iterative Message Passing”, Art 84 arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Jan. 10, 2017 (Jan. 10, 2017), XP080740455, DO… [cited by applicant]
Yuren Con et al., “Spatial-Temporal Transformer for Dynamic Scene Graph Generation”, 18th IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021, pp. 16352-16362, ISBN (Electronic) 978-1-6654-2812-5,… [cited by applicant]
Charles R Qi et a., “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space”, Jun. 2017, arXiv:1706.02413 [cs.CV]. [cited by applicant]
Satoshi Tsutsui et al., “Scene Graph Generation from Images”, 2016. [cited by applicant]
Yikang Li et al., “Scene Graph Generation from Objects, Phrases and Region Captions”, 2017 IEEE International Conference on Computer Vision (ICCV), Jul. 31, 2017, DOI:10.1109/ICCV.2017.142, arXiv:1707.09700 [cs.CV]. [cited by applicant]
Angela Dai et al, “ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21-26, 2017, DOI: 10.1109/CVPR.2017.261, arXiv:1702.04405 [… [cited by applicant]
Yin Zhou et al., “VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection”, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 18-23, 2018, DOI: 10.1109/CVPR.2018.00472, arXiv:1711… [cited by applicant]
Sekunde, “3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (CVPR2019 Oral)”, https://github.com/Sekunde/3D-SIS#Overview. [cited by applicant]
Ximing Yang et al., “Generate Point Clouds with Multiscale Details from Graph-Represented Structures”, arXiv:2112.06433 [cs.CV], Mar. 9, 2022. [cited by applicant]
Hugues Thomas et al., “KPConv: Flexible and Deformable Convolution for Point Clouds”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27, 2019-Nov. 2, 2019, DOI: 10.1109/ICCV.2019.00651, arXiv:190… [cited by applicant]
Johanna Wald et al., “Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions (CVPR 2020)”, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 13-19, 2020, DOI: 10.1109/CVPR42600.2… [cited by applicant]
Dong Wook Shu et al., “3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions”, IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3859-3868, arXiv:1905.06292 [cs.C… [cited by applicant]
Nuscenes By Motional, https://www.nuscenes.org/nuscenes. [cited by applicant]
SemanticKITTI—A Dataset for LiDAR-based Semantic Scene Understanding, http://semantic-kitti.org. [cited by applicant]
Xiaojun Chang et al., “Scene Graphs: A Survey of Generations and Applications”, DOI: 10.48550/arXiv.2104.01111, arXiv:2104.01111 [cs.CV]. [cited by applicant]
Dilara Gokay et al., “Graph2Pix: A Graph-Based Image to Image Translation Framework”, Technische Universität München Boǧaziçi University, arXiv:2108.09752 [cs.CV]. [cited by applicant]