IP Library › Granted Patent US 12,222,832
Granted Patent B2
US 12,222,832 · App. 18/466,286 · Granted Feb 11, 2025

Systems and methods for generating synthetic sensor data via machine learning

Inventors: Sivabalan Manivasagam (Toronto, CA); Shenlong Wang (Toronto, CA); Wei-Chiu Ma (Toronto, CA); Kelvin Ka Wing Wong (Toronto, CA); Wenyuan Zeng (Toronto, CA); Raquel Urtasun (Toronto, CA)
Assignee: AURORA OPERATIONS, INC.
G06F11/263G06F17/18G06N3/047G06N20/00G06T15/06G06T17/20G06T2207/10028
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,222,832
App. No.
18/466,286
Granted
Feb 11, 2025
Kind
B2
Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Claims (76)

1. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing:

one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:

using a physics-based simulation engine to obtain an initial synthetic point cloud,

generating, by a machine-learned model, a value corresponding to a probability that a real LIDAR point cloud would have an error at a point in the initial synthetic point cloud, and

generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and

instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:

obtaining LiDAR data descriptive of an environment of the autonomous vehicle;

determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and

controlling the autonomous vehicle according to the motion plan.

2. The autonomous vehicle control system of claim 1 , wherein generating, based on the value, the synthetic LIDAR data to contain the error at the point comprises:

sampling, based on the value, to determine whether to include the error at the point in the synthetic LIDAR data.

3. The autonomous vehicle control system of claim 1 , wherein the probability corresponds to a dropout probability.

4. The autonomous vehicle control system of claim 3 , wherein generating the synthetic LIDAR data to contain the error at the point comprises omitting the point.

5. The autonomous vehicle control system of claim 1 , wherein the synthetic LIDAR data was generated by:

generating, using the machine-learned model, a probability mask over the initial synthetic point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and

sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

6. The autonomous vehicle control system of claim 1 , wherein the at least one of the one or more machine-learned models was tested by:

processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

7. The autonomous vehicle control system of claim 6 , wherein the at least one of the one or more machine-learned models was tested by:

generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and

processing the additional synthetic LiDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

8. The autonomous vehicle control system of claim 6 , wherein the at least one of the one or more machine-learned models was tested by:

inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.

9. One or more non-transitory computer-readable media storing:

one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:

using a physics-based simulation engine to obtain an initial synthetic point cloud,

generating, by a machine-learned model, a value corresponding to a probability that a real LiDAR point cloud would have an error at a point in the initial synthetic point cloud, and

generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and

instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising:

obtaining LIDAR data descriptive of an environment of the autonomous vehicle;

determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and

controlling the autonomous vehicle according to the motion plan.

10. The one or more non-transitory computer-readable media of claim 9 , wherein using the machine-learned model to add realistic errors to the initial synthetic point cloud comprises:

sampling, based on the value, to determine whether to include the error at the point in the synthetic LIDAR data and.

11. The one or more non-transitory computer-readable media of claim 9 , wherein the probability corresponds to a dropout probability.

12. The one or more non-transitory computer-readable media of claim 11 , wherein generating, based on the sampling, the synthetic LIDAR data to contain the error at the point comprises omitting the point.

13. The one or more non-transitory computer-readable media of claim 9 , wherein using the machine-learned model to add realistic errors to the initial synthetic point cloud comprises:

generating, using the machine-learned model, a probability mask over the initial synthetic point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and

sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

14. The one or more non-transitory computer-readable media of claim 9 , wherein the at least one of the one or more machine-learned models was tested by:

processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

15. The one or more non-transitory computer-readable media of claim 14 , wherein the at least one of the one or more machine-learned models was tested by:

generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and

processing the additional synthetic LiDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

16. The one or more non-transitory computer-readable media of claim 14 , wherein the at least one of the one or more machine-learned models was tested by:

inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.

17. A method, comprising:

obtaining LiDAR data descriptive of an environment of an autonomous vehicle;

determining, using one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:

using a physics-based simulation engine to obtain an initial synthetic point cloud,

generating, by a machine-learned model, a value corresponding to a probability that a real LiDAR point cloud would have an error at a point in the initial synthetic point cloud,

generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and

controlling the autonomous vehicle according to the motion plan.

18. The method of claim 17 , wherein using the machine-learned model to add realistic errors to the initial point cloud comprises:

generating, using the machine-learned model, a probability mask over the initial point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and

sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

19. The method of claim 17 , wherein the at least one of the one or more machine-learned models was tested by:

processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

20. The method of claim 19 , wherein the at least one of the one or more machine-learned models was tested by:

generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and

processing the additional synthetic LIDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

21. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing:

one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:

using a physics-based simulation engine to obtain an initial synthetic point cloud, and

using a machine-learned model to add realistic errors to the initial synthetic point cloud to obtain the synthetic LiDAR data;

wherein the at least one of the one or more machine-learned models was tested by processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data; and

instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:

obtaining LiDAR data descriptive of an environment of the autonomous vehicle;

determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and

controlling the autonomous vehicle according to the motion plan.

22. The autonomous vehicle control system of claim 21 , wherein the at least one of the one or more machine-learned models was tested by:

inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: UATC, LLC
To: AURORA OPERATIONS, INC.
Reel/Frame 067733/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2024
From: MANIVASAGAM, SIVABALAN; MA, WEI-CHIU; ZENG, WENYUAN; WONG, KELVIN KA WING; WANG, SHENLONG
To: UATC, LLC
Reel/Frame 066928/0729 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2024
From: URTASUN, RAQUEL
To: UATC, LLC
Reel/Frame 066928/0741 →
Continuity (6)
Continuation 17727085 · Apr 22, 2022
Continuation 16826990 · Mar 23, 2020
Provisional Application 62950279 · Dec 19, 2019
Provisional Application 62936439 · Nov 16, 2019
Provisional Application 62822844 · Mar 23, 2019
Related Publication 20230418717A1 · Dec 28, 2023
References Cited (62)
US 10699167B1 · Dowdall et al. · 2020 [cited by applicant]
US 10839530B1 · Berger · 2020 [cited by examiner]
US 11544167B2 · Manivasagam et al. · 2023 [cited by applicant]
US 20170200309A1 · Qian et al. · 2017 [cited by applicant]
US 20190096125A1 · Schulter et al. · 2019 [cited by applicant]
US 20190303759A1 · Farabet · 2019 [cited by examiner]
US 20200110158A1 · Ecins et al. · 2020 [cited by applicant]
US 20200118281A1 · Kwon et al. · 2020 [cited by applicant]
U.S. Appl. No. 62/648,399, filed Mar. 27, 2018 (Year: 2018). [cited by examiner]
Alhaija, “Augmented Reality Meets Computer Vision”, International Journal of Computer Vision, 1-12. [cited by applicant]
Bai et al., “Deep multi-sensor lane detection.” In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2018, pp. 3102-3109. [cited by applicant]
Besl et al., “Method for Registration of 3-D Shapes”, Transactions on Pattern Analysis and Machine Intelligence, vol. 14, No. 2, Feb. 1992, pp. 239-256. [cited by applicant]
Butler, “A Naturalistic Open Source Movie for Optical Flow Evaluation”, European Conference on Computer Vision, 2012, 611-625. [cited by applicant]
Chen, “Multi-View 3D Object Detection Network for Autonomous Driving”, arXiv:1611.07759v3, Jun. 22, 2017, 9 pages. [cited by applicant]
Choy et al., “3D-R2N2: AU nified Approach for Single and Multi-View 3D Object Reconstruction”, Conference on Computer Vision, Jun. 26-Jul. 1, 2016, Las Vegas, Nevada, 17 pages. [cited by applicant]
Deschaud, “Automatic data driven vegetation modeling for lidar simulation”, ICRA, May 2012, pp. 5030-5036. [cited by applicant]
Dewan, “Rigid Scene Flow for 3D LiDAR Scans”, IROS, 2016, pp. 1-6. [cited by applicant]
Dosovitskiy, “CARLA: An Open Urban Driving Simulator”, 1st Conference on Robot Learning, 2017, Mountain View, United States, pp. 1-16. [cited by applicant]
Epic Games, “Unreal Engine”, https://www.unrealengine.com/, retrieved Oct. 20, 2020, 7 pages. [cited by applicant]
Fang et al., “Simulating LIDAR Point Cloud for Autonomous Driving using Real-world Scenes and Traffic Flows”, Nov. 2018, 7 pages. [cited by applicant]
Fang, “Augmented LiDAR Simulator for Autonomous Driving”, arXiv:1811.07112v2, 10 pages. [cited by applicant]
Gaidon, “Virtual Worlds as Proxy for Multi-Object Tracking Analysis”, arXiv:1605.06457v1, 10 pages. [cited by applicant]
Gschwandtner, “BlenSor: Blender Sensor Simulation Toolbox”, International Conference on Advances in Visual Computing, Date, Berlin, Heidel-berg, pp. 199-208. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2020/024169, mailed Sep. 4, 2020, 17 pages. [cited by applicant]
Johnson, “The Malmo Platforrn for Artificial Intelligence Experimentation.” Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, Jul. 2016, 2 pages. [cited by applicant]
Kempka, “ViZDoom: A Doom-based AI Research Platforrn for Visual Reinforcement Learning”, arXiv:1605.02097V2. 8 pages. [cited by applicant]
Luo, Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net, Jun. 18-22, 2018, Salt Lake City, pp. 3569-3577. [cited by applicant]
Maturana, “VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition”, IEEE/RSJ International Conference on Intelligent Robots and Systems, Sep. 28-Oct. 2, 2015, Hamburg, Germany, pp. 922-928. [cited by applicant]
Mayer, “A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation”. arXiv:1512.02134V1. Dec. 7, 2015. 14 pages. [cited by applicant]
Mnih, “Playing Atari with Deep Reinforcement Learning”, arXiv:1312.5602v1, Dec. 19, 2013, 9 pages. [cited by applicant]
Moller, “Fast Minimum Storage Ray Triangle Intersection”, Journal of Graphics Tools, vol. 2., Issue 1., Apr. 6, 2012, pp. 21-28. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/567,607, mailed May 13, 2022, 9 pages. [cited by applicant]
Pomerlaeu, “Alvinn: An Autonomous Land Vehicle In A Neural Network”, NIPS, 1988, pp. 305-313. [cited by applicant]
Provisional Application of Farabet et al. U.S. Appl. No. 62/648,399, filed Mar. 27, 2018 (Year:2018). [cited by applicant]
Qi, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, arXiv:1612.00593v2, Apr. 10, 2017, 19 pages. [cited by applicant]
Qi et al, “Volumetric and Multi-View CNNs for Object Classification on 3D Data”, arXiv: 1604V2, Apr. 29, 2016, 14 pages. [cited by applicant]
Reigler, “OctNet: Learning Deep 3D Representations at High Resolutions”, arXiv:1611.05009v4, Apr. 10, 2017. 21 pages. [cited by applicant]
Richter, “Playing for Benchmarks”, arXiv:1709.07322V1, Sep. 21, 2017, 10 pages. [cited by applicant]
Ronneberger, “U-Net: Convolutional Networks for Biomedical Image Segmentation”, arXiv:1505.04597v1, May 18, 2015, 8 pages. [cited by applicant]
Ros, “The SYNTHIA Dataset: A large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes”, IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27-30, 2016, Las Vegas, Nevada, pp. 3234-324… [cited by applicant]
Shrivastava, “Learning from Simulated and Unsupervised Images through Adversarial Training”, IEEE Conference on Computer Vision and Pattern Recognition, Jul. 21-26, Honolulu, Hawaii, pp. 2107-2116. [cited by applicant]
Song, “Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 27-30, 2016. Las Vevas, Nevada. pp. 1-9. [cited by applicant]
Sun, “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume”, arXiv:1709.02371v3, Jun. 25, 2018, 18 pages. [cited by applicant]
Tan, “Sim-to-Real: Learning Agile Locomotion For Quadruped Robots”, arXiv:1804.10332v2, May 16, 2018, 11 pages. [cited by applicant]
Thrun, “The Graph SLAM Algorithm with Applications to Large-Scale Mapping of Urban Structures”, 2 The International Journal of Robotics Research, vol. 25, No, 5-6, May-Jun. 2006, pp. 403-429. [cited by applicant]
Todorov, “MuJoCo: A physics engine for model-based control”, IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 7, 2010, Algarce, Portugal, pp. 1-8. [cited by applicant]
TORCS, “The Open Racing Car Simulator”, http://torcs.sourceforgenet, retrieved on Jul. 30, 2019, 4 pages. [cited by applicant]
Unity, “Unity game engine-official site”,2008, http://unity3d.com, retrieved on Oct. 20, 2020, 13 pages. [cited by applicant]
Unity “Performance By Default, Next-Gen Rendering and Artist Tooling”, https://unity3d.com, retrieved on Jul. 31, 2019, 12 pages. [cited by applicant]
Unreal Engine, https://unrealengine.com, retrieved on Jul. 31, 2019, 8 pages. [cited by applicant]
Velodyne Manual “High Definition LiDAR Sensor”, Velodyne, Inc., Morgan Hill, California, 2013, 54 pages. [cited by applicant]
Velodyne LiDAR, Inc., “User Manual”. San Jose, CA, 2018, 146 pages [cited by applicant]
Wang, “Deep Parametric Continuous Convolutional Neural Networks”, CVPR, Jun. 18-21, 2018, Salt Lake City, Utah, pp. 2589-2597. [cited by applicant]
Wu et al., “SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaption for Road-Object Segmentation from a LiDAR Point Cloud”, arXiv:1809.08495v1, 7 pages. [cited by applicant]
Xia, “Gibson Env: Real-World Perception for Embodied Agents”, arXiv:1808.10654v1, Aug. 31, 2018, 12 pages. [cited by applicant]
Yang et al., “PIX OR: Real-Time 3D Object Detection from Point Clouds”, Conference on Computer Vision and Pattern Recognition, Jun. 18-22, 2018, Salt Lake City, Utah, pp. 7652-7660. [cited by applicant]
Yi, “SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation”, arXiv:1612.00606v1, Dec. 2, 2016, 11 pages. [cited by applicant]
Yue et al., “A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving”, arXiv:1804.00103v1, 7 pages. [cited by applicant]
Zhang, “Efficient Convolutions for Real-Time Semantic Segmentation of 3D Point Clouds”, International Conference on 3D Vision (3DV), Sep. 5-8, 2018, Verona, Italy, pp. 1-10. [cited by applicant]
Zhang et al., “Road extraction by deep residual u-net.” [cited by applicant]
Zhou, “Open3D: A Modern Library for 3D Data Processing”, arXiv:1801.09847v1, Jan. 30, 2018, 6 pages. [cited by applicant]
Zhu, “Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning”, arXiv:1609.05143v1, Sep. 16, 2016, 8 pages. [cited by applicant]