IP Library › Granted Patent US 11,741,661
Granted Patent B2
US 11,741,661 · App. 17/321,205 · Granted Aug 29, 2023

Sensor simulation with unified multi-sensor views

Inventors: Anas Lasram (South San Francisco, CA); James Graham Dolan (Foster City, CA)
Assignee: Zoox, Inc.
G06T15/10G06T15/06G07C5/0808
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,741,661
App. No.
17/321,205
Granted
Aug 29, 2023
Kind
B2
Abstract

A sensor simulation system may generate sensor data for use in simulations by rendering two-dimensional views of a three-dimensional simulated environment. In various examples, the sensor simulation system uses sensor dependency data to determine specific views to be re-rendered at different times during the simulation. The sensor simulation system also may generate unified views with multi-sensor data at each region (e.g., pixel) of the two-dimensional view for consumption by different sensor types. A hybrid technique may be used in some implementations in which rasterization is used to generate a view, after which ray tracing is used to align the view with a particular sensor. Spatial and temporal upsampling techniques also may be used, including depth-aware and velocity-aware analyses for simulated objects, to improve view resolution and reduce the frequency of re-rendering views.

Claims (91)

1. A method comprising:

receiving data representing a three-dimensional simulated environment;

generating a two-dimensional view of the three-dimensional simulated environment;

determining a first metadata and second metadata associated with a region of the two-dimensional view;

determining first sensor data based at least in part on the two-dimensional view, wherein the first sensor data includes the first metadata;

providing the first sensor data to a first simulated sensor configured to detect the first metadata;

determining second sensor data based at least in part on the two-dimensional view, wherein the second sensor data includes the second metadata; and

providing the second sensor data to a second simulated sensor configured to detect the second metadata.

2. The method of claim 1 , wherein:

determining the first sensor data comprises generating a first reprojection of the two-dimensional view based at least in part on a first sensor location of the first simulated sensor within the three-dimensional simulated environment; and

determining the second sensor data comprises generating a second reprojection of the two-dimensional view based at least in part on a second sensor location of the second simulated sensor within the three-dimensional simulated environment, wherein the first sensor location is different from the second sensor location.

3. The method of claim 1 , wherein:

generating the first sensor data comprises generating a first reprojection of the two-dimensional view based at least in part on a first simulated sensor time associated with the first simulated sensor; and

generating the second sensor data comprises generating a second reprojection of the two-dimensional view based at least in part on a second simulated sensor time associated with the second simulated sensor, wherein the first simulated sensor time is different from the second simulated sensor time.

4. The method of claim 1 , wherein:

the first simulated sensor includes a camera, and the first metadata includes light data associated with the region of the two-dimensional view; and

the second simulated sensor includes at least one of a lidar sensor or a radar sensor, and the second metadata includes at least one of velocity data or material data associated with the region of the two-dimensional view.

5. The method of claim 1 , wherein generating the first sensor data comprises:

determining a first sensor location associated with the first simulated sensor within the three-dimensional simulated environment; and

generating a reprojection of the two-dimensional view, based at least in part on a difference between the first sensor location and a view focal point location associated with the two-dimensional view,

wherein generating the two-dimensional view comprises performing a rasterization process, and

wherein generating the reprojection of the two-dimensional view comprises performing a ray casting process.

6. The method of claim 1 , wherein determining the first sensor data comprises:

determining a first simulated sensor time associated with the first simulated sensor; and

generating a reprojection of the two-dimensional view, based at least in part on a time difference between the first simulated sensor time and a time associated with generating the two-dimensional view,

wherein generating the two-dimensional view comprises performing a rasterization process, and

wherein generating the reprojection of the two-dimensional view comprises performing a ray casting process.

7. The method of claim 6 , further comprising:

determining a first movement of the first simulated sensor within the three-dimensional simulated environment, during a time period associated with the time difference; and

determining a second movement of a simulated object represented within the two-dimensional view, during the time period,

wherein generating the reprojection of the two-dimensional view is further based at least in part on the first movement and the second movement.

8. The method of claim 1 , wherein generating the two-dimensional view comprises:

storing camera data at a first storage location associated with the region;

storing lidar data at a second storage location associated with the region;

storing radar data at a third storage location associated with the region; and

storing an object identifier at a fourth storage location associated with the region.

9. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:

receiving data representing a three-dimensional simulated environment;

generating a two-dimensional view of the three-dimensional simulated environment;

determining a first metadata and second metadata associated with a region of the two-dimensional view;

determining first sensor data based at least in part on the two-dimensional view, wherein the first sensor data includes the first metadata;

providing the first sensor data to a first simulated sensor configured to detect the first metadata;

determining second sensor data based at least in part on the two-dimensional view, wherein the second sensor data includes the second metadata; and

providing the second sensor data to a second simulated sensor configured to detect the second metadata.

10. The one or more non-transitory computer-readable media of claim 9 , wherein:

determining the first sensor data comprises generating a first reprojection of the two-dimensional view based at least in part on a first sensor location of the first simulated sensor within the three-dimensional simulated environment; and

determining the second sensor data comprises generating a second reprojection of the two-dimensional view based at least in part on a second sensor location of the second simulated sensor within the three-dimensional simulated environment, wherein the first sensor location is different from the second sensor location.

11. The one or more non-transitory computer-readable media of claim 9 , wherein:

generating the first sensor data comprises generating a first reprojection of the two-dimensional view based at least in part on a first simulated sensor time associated with the first simulated sensor; and

generating the second sensor data comprises generating a second reprojection of the two-dimensional view based at least in part on a second simulated sensor time associated with the second simulated sensor, wherein the first simulated sensor time is different from the second simulated sensor time.

12. The one or more non-transitory computer-readable media of claim 9 , wherein:

the first simulated sensor includes a camera, and the first metadata includes light data associated with the region of the two-dimensional view; and

the second simulated sensor includes at least one of a lidar sensor or a radar sensor, and the second metadata includes at least one of velocity data or material data associated with the region of the two-dimensional view.

13. The one or more non-transitory computer-readable media of claim 9 , wherein generating the first sensor data comprises:

determining a first sensor location associated with the first simulated sensor within the three-dimensional simulated environment; and

generating a reprojection of the two-dimensional view, based at least in part on a difference between the first sensor location and a view focal point location associated with the two-dimensional view,

wherein generating the two-dimensional view comprises performing a rasterization process, and

wherein generating the reprojection of the two-dimensional view comprises performing a ray casting process.

14. The one or more non-transitory computer-readable media of claim 9 , wherein determining the first sensor data comprises:

determining a first simulated sensor time associated with the first simulated sensor; and

generating a reprojection of the two-dimensional view, based at least in part on a time difference between the first simulated sensor time and a time associated with generating the two-dimensional view,

wherein generating the two-dimensional view comprises performing a rasterization process, and

wherein generating the reprojection of the two-dimensional view comprises performing a ray casting process.

15. The one or more non-transitory computer-readable media of claim 14 , the operations further comprising:

determining a first movement of the first simulated sensor within the three-dimensional simulated environment, during a time period associated with the time difference; and

determining a second movement of a simulated object represented within the two-dimensional view, during the time period,

wherein generating the reprojection of the two-dimensional view is further based at least in part on the first movement and the second movement.

16. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:

receiving data representing a three-dimensional simulated environment;

generating a two-dimensional view of the three-dimensional simulated environment;

determining a first metadata and second metadata associated with a region of the two-dimensional view;

determining first sensor data based at least in part on the two-dimensional view, wherein the first sensor data includes the first metadata;

providing the first sensor data to a first simulated sensor configured to detect the first metadata;

determining second sensor data based at least in part on the two-dimensional view, wherein the second sensor data includes the second metadata; and

providing the second sensor data to a second simulated sensor configured to detect the second metadata.

17. The system of claim 16 , wherein:

determining the first sensor data comprises generating a first reprojection of the two-dimensional view based at least in part on a first sensor location of the first simulated sensor within the three-dimensional simulated environment; and

determining the second sensor data comprises generating a second reprojection of the two-dimensional view based at least in part on a second sensor location of the second simulated sensor within the three-dimensional simulated environment, wherein the first sensor location is different from the second sensor location.

18. The system of claim 16 , wherein:

generating the first sensor data comprises generating a first reprojection of the two-dimensional view based at least in part on a first simulated sensor time associated with the first simulated sensor; and

generating the second sensor data comprises generating a second reprojection of the two-dimensional view based at least in part on a second simulated sensor time associated with the second simulated sensor, wherein the first simulated sensor time is different from the second simulated sensor time.

19. The system of claim 16 , wherein:

the first simulated sensor includes a camera, and the first metadata includes light data associated with the region of the two-dimensional view; and

the second simulated sensor includes at least one of a lidar sensor or a radar sensor, and the second metadata includes at least one of velocity data or material data associated with the region of the two-dimensional view.

20. The system of claim 16 , wherein generating the first sensor data comprises:

determining a first sensor location associated with the first simulated sensor within the three-dimensional simulated environment; and

generating a reprojection of the two-dimensional view, based at least in part on a difference between the first sensor location and a view focal point location associated with the two-dimensional view,

wherein generating the two-dimensional view comprises performing a rasterization process, and

wherein generating the reprojection of the two-dimensional view comprises performing a ray casting process.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 14, 2021
From: LASRAM, ANAS; DOLAN, JAMES GRAHAM
To: ZOOX, INC.
Reel/Frame 056251/0448 →
Continuity (1)
Related Publication 20220366636A1 · Nov 17, 2022