Capturing and simulating radar data for autonomous driving systems
A simulation system may generate radar data for synthetic simulations of autonomous vehicles, by using a data store of object radar data and attributes determined from sensor data captured in real-world physical environments. The radar data store may include radar point clouds representing real-world objects and associated object attributes, as well as radar background data captured for a number of physical environments. The simulation system may construct radar data for use in a simulation based on radar object data and/or radar background data, including using various probabilities within various overlay regions to determine subsets of object and background radar points to be rendered. During a simulation, the generated radar data may be provided to a simulated radar sensor of a simulated vehicle configured to execute trained perception models based on radar data input.
1 . One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving simulation data representing a simulated environment associated with a simulation;
determining, based at least in part on the simulation data, a simulated object within the simulated environment and a first attribute associated with the simulated object;
determining, using a data store and based at least in part on the first attribute, that a non-simulated object represented in sensor data corresponds to the simulated object, wherein the sensor data comprises data from a vehicle operating in a real environment;
determining, from the data store and based at least in part on the first attribute, object simulation data associated with the non-simulated object , wherein the object simulation data includes:
a second attribute associated with the non-simulated object, wherein the second attribute corresponds at least in part to the first attribute; and
a plurality of radar points associated with the non-simulated object, wherein the plurality of radar points excludes other radar data from a scene of the real environment from which the plurality of radar points was captured;
retrieving the plurality of radar points from the data store;
generating simulation radar data based at least in part on the plurality of radar points; and
rendering the simulated object during the simulation, wherein rendering the simulated object includes providing the simulation radar data as input to a simulated radar sensor.
2 . The one or more non-transitory computer-readable media of claim 1 , wherein generating the simulation radar data comprises:
determining a first radar point in the plurality of radar points; and
modifying the first radar point to generate a second radar point, wherein the modifying includes modifying at least one of a radar cross-section value or a velocity value of the first radar point.
3 . The one or more non-transitory computer-readable media of claim 1 , wherein the plurality of radar points includes:
a first radar point of the sensor data, within a bounding box associated with the non-simulated object; and
a second radar point of the sensor data, within a buffer region around the bounding box.
4 . The one or more non-transitory computer-readable media of claim 1 , wherein generating the simulation radar data comprises:
determining a difference between the first attribute and the second attribute; and
applying a linear transformation to the plurality of radar points, based at least in part on the difference.
5 . The one or more non-transitory computer-readable media of claim 1 , wherein the first attribute associated with the simulated object comprises at least one of:
a range of the simulated object;
an azimuth of the simulated object;
a yaw of the simulated object;
an object classification of the simulated object;
a width of a bounding box associated with the simulated object; or
a height of a bounding box associated with the simulated object.
6 . The one or more non-transitory computer-readable media of claim 1 , wherein the first attribute and the second attribute include at least one of a relative object position or a relative object angle, and wherein determining the object simulation data comprises determining that a difference between the first attribute and the second attribute is less than a threshold.
7 . The one or more non-transitory computer-readable media of claim 1 , the operations further comprising:
determining, based at least in part on the simulation data, at least one of:
a first object classification associated with the simulated object;
a first angular velocity associated with the simulated object; or
a first occlusion associated with the simulated object, and
wherein the non-simulated object includes at least one of:
a second object classification different from the first object classification;
a second angular velocity different from the first angular velocity; or
a second occlusion different from the first occlusion.
8 . A system comprising:
a radar point cloud data store;
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving simulation data representing a simulated environment associated with a simulation;
determining, based at least in part on the simulation data, a simulated object within the simulated environment and a first attribute associated with the simulated object;
determining, using the radar point cloud data store and based at least in part on the first attribute, that a non-simulated object represented in sensor data corresponds to the simulated object, wherein the sensor data comprises data from a vehicle operating in a real environment;
determining, from the radar point cloud data store and based at least in part on the first attribute, object simulation data associated with the non-simulated object, wherein the object simulation data includes:
a second attribute associated with the non-simulated object, wherein the second attribute corresponds at least in part to the first attribute; and
a plurality of radar points associated with the non-simulated object, wherein the plurality of radar points excludes other radar data from a scene of the real environment from which the plurality of radar points was captured;
retrieving the plurality of radar points from the radar point cloud data store;
generating simulation radar data based at least in part on the plurality of radar points; and
rendering the simulated object during the simulation, wherein rendering the simulated object includes providing the simulation radar data as input to a simulated radar sensor.
9 . The system of claim 8 , wherein the radar point cloud data store stores a plurality of object entries, wherein an object entry of the plurality of object entries includes:
a range of an object associated with the object entry;
an azimuth of an object associated with the object entry;
a yaw of an object associated with the object entry;
a doppler value of an object associated with the object entry;
a width of an object associated with the object entry; and
a height of an object associated with the object entry.
10 . The system of claim 8 , wherein storing the object simulation data comprises:
determining a first plurality of radar points in the radar point cloud; and
modifying individual radar points of the first plurality of radar points to generate a second plurality of radar points, wherein the modifying includes modifying at least one of a radar cross-section value or a velocity value of the individual radar points.
11 . The system of claim 8 , wherein determining the radar point cloud associated with the object comprises:
determining a bounding box associated with the object, based at least in part on the sensor data;
determining a buffer region around the bounding box; and
determining, as the radar point cloud, a set of radar data points within the bounding box and the buffer region.
12 . The system of claim 8 , the operations further comprising:
inputting, during the simulation, the radar point cloud to a trained neural network configured to detect objects based on the sensor data.
13 . A method comprising:
receiving simulation data representing a simulated environment associated with a simulation;
determining, based at least in part on the simulation data, a simulated object within the simulated environment and a first attribute associated with the simulated object;
determining, using a data store and based at least in part on the first attribute, that a non-simulated object represented in sensor data corresponds to the simulated object, wherein the sensor data comprises data from a vehicle operating in a real environment;
determining, from the data store and based at least in part on the first attribute, object simulation data associated with the non-simulated object, wherein the object simulation data includes:
a second attribute associated with the non-simulated object, wherein the second attribute corresponds at least in part to the first attribute; and
a plurality of radar points associated with the non-simulated object, wherein the plurality of radar points excludes other radar data from a scene of the real environment from which the plurality of radar points was captured;
retrieving the plurality of radar points from the data store;
generating simulation radar data based at least in part on the plurality of radar points; and
rendering the simulated object during the simulation, wherein rendering the simulated object includes providing the simulation radar data as input to a simulated radar sensor.
14 . The method of claim 13 , wherein determining the plurality of radar points comprises:
determining a first radar point in the plurality of radar points;
modifying the first radar point to generate a modified radar point, wherein the modifying includes modifying at least one of a radar cross-section value or a velocity value of the first radar point; and
replacing the first radar point in the plurality of radar points with the modified radar point.
15 . The method of claim 13 , wherein determining the plurality of radar points associated with the non-simulated object comprises at least one of:
executing a density-based clustering algorithm on the sensor data; or
determining a set of radar data points within a bounding box associated with the non-simulated object.
16 . The method of claim 13 , wherein determining the plurality of radar points associated with the non-simulated object comprises:
determining a bounding box associated with the non-simulated object, based at least in part on the sensor data;
determining a buffer region around the bounding box; and
determining, as the plurality of radar points, a set of radar data points within the bounding box and the buffer region.
17 . The method of claim 13 , wherein at least one of the first attribute or the second attribute comprises at least one of:
a range of the respective simulated or non-simulated object;
an azimuth of the respective simulated or non-simulated object;
a yaw of the respective simulated or non-simulated object;
an object classification of the respective simulated or non-simulated object;
a width of a bounding box associated with the respective simulated or non-simulated object; or
a height of a bounding box associated with the respective simulated or non-simulated object.
18 . The method of claim 13 , wherein the first attribute and the second attribute include at least one of a relative object position or a relative object angle, and wherein determining the object simulation data comprises determining that a difference between the first attribute and the second attribute is less than a threshold.
19 . The method of claim 13 , further comprising:
determining, based at least in part on the sensor data, at least one of:
a first object classification associated with the non-simulated object;
a first angular velocity associated with the non-simulated object; or
a first occlusion associated with the non-simulated object,
wherein the simulated object includes at least one of:
a second object classification different from the first object classification;
a second angular velocity different from the first angular velocity; or
a second occlusion different from the first occlusion.
20 . The method of claim 13 , the operations further comprising:
inputting, during the simulation, the radar point cloud to a trained neural network configured to detect objects based on the sensor data.