Systems and methods for simulating sensor data using a generative model

1. A simulation system, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing:

an input module including instructions that when executed by the one or more processors cause the one or more processors to, in response to receiving a request to generate simulated sensor data for a scene, acquire simulation data about the scene, wherein the simulation data includes at least simulated information about the scene that is computer-generated; and

a simulation module including instructions that, when executed by the one or more processors, cause the one or more processors to compute the simulated sensor data using a generative neural network that accepts the simulation data as an input and produces the simulated sensor data as an output, wherein the simulated sensor data is a simulated perception of the scene by a sensor, wherein the generative neural network includes a first generative network for producing the simulated sensor data, and a second generative network for inferring semantic labels,

wherein the simulation module includes instructions to train the generative neural network by computing at least a cycle consistency loss that indicates a transitive relationship between learned mappings of the first generative network and the second generative network, and that includes a forward cycle consistency and a backward cycle consistency.

2. The simulation system of claim 1 , wherein the simulated data about the scene includes semantic labels identifying objects depicted in the scene and ranges of the objects from a perspective of the sensor,

wherein the simulation data simulates at least range information associated with the scene as perceived by at least one of: an image sensor and a light detection and ranging (LiDAR) sensor, and

wherein the simulated sensor data is intensity data that indicates intensities of reflected light from observed points on the objects, and wherein the intensities correspond with one or more attributes of the objects.

3. The simulation system of claim 1 , wherein the input module includes instructions to acquire the simulation data including instructions to simulate the scene using a rendering engine that produces the scene as a three-dimensional environment including objects and spatial relationships between the objects to produce the simulation data, and

wherein the simulation data includes semantic labels and range data for the scene.

4. The simulation system of claim 1 , wherein the simulation module includes instructions to provide the simulated sensor data including instructions to generate an intensity map and integrate the intensity map with a simulation for the scene to correlate the simulated sensor data with objects in the scene, and

wherein the simulation including the scene is computer-generated and provides inputs for training one or more machine learning tasks.

5. The simulation system of claim 1 , wherein the generative neural network is a generative adversarial network (GAN) that includes the first generative network for producing the simulated sensor data, a first discriminator, the second generative network for inferring semantic labels, and a second discriminator, and

wherein the simulation module includes instructions to compute the simulated sensor data using the generative neural network including using learned correlations embodied within the first generative network to infer the simulated sensor data from the simulation data.

6. The simulation system of claim 5 , wherein the simulation module includes instructions to train the generative neural network using an unsupervised learning process that includes using the first discriminator to assess first output data of the first generative network in comparison with training data and using the second discriminator to assess second output data of the second generative network in comparison with the training data, the training data including training sensor data and training semantic labels associated with at least one training scene.

7. The simulation system of claim 6 , wherein the simulation module includes instructions to train including instructions to iteratively compute a loss according to a loss function and adjusting parameters of the generative neural network according to the loss, and

wherein the simulation module includes instructions to iteratively compute the loss including instructions to compute an adversarial loss and the cycle consistency loss, the adversarial loss indicating at least an objective loss associated with the first discriminator identifying whether the first output data is computer generated.

8. A non-transitory computer-readable medium including instructions that when executed by one or more processors cause the one or more processors to:

in response to receiving a request to generate simulated sensor data for a scene, acquire simulation data about the scene, wherein the simulation data includes at least simulated information about the scene that is computer-generated;

compute the simulated sensor data using a generative neural network that accepts the simulation data as an input and produces the simulated sensor data as an output, wherein the generative neural network includes a first generative network for producing the simulated sensor data, and a second generative network for inferring semantic labels, wherein the simulated sensor data is a simulated perception of the scene by a sensor; and

train the generative neural network by computing at least a cycle consistency loss that indicates a transitive relationship between learned mappings of the first generative network and the second generative network, and that includes a forward cycle consistency and a backward cycle consistency.

9. The non-transitory computer-readable medium of claim 8 , wherein the simulated data about the scene includes semantic labels identifying objects depicted in the scene and ranges of the objects from a perspective of the sensor,

wherein the simulation data simulates at least range information associated with the scene as perceived by at least one of: an image sensor and a light detection and ranging (LiDAR) sensor, and

wherein the simulated sensor data is intensity data that indicates intensities of reflected light from observed points on the objects, and wherein the intensities correspond with one or more attributes of the objects.

10. The non-transitory computer-readable medium of claim 8 , wherein the instructions to acquire the simulation data include instructions to simulate the scene using a rendering engine that produces the scene as a three-dimensional environment including objects and spatial relationships between the objects to produce the simulation data, and

wherein the simulation data includes semantic labels and range data for the scene.

11. The non-transitory computer-readable medium of claim 8 , wherein the instructions to provide the simulated sensor data include instructions to generate an intensity map and integrate the intensity map with a simulation for the scene to correlate the simulated sensor data with objects in the scene, and

wherein the simulation including the scene is computer-generated and provides inputs for training one or more machine learning tasks.

12. The non-transitory computer-readable medium of claim 8 , wherein the generative neural network is a generative adversarial network (GAN) that includes the first generative network for producing the simulated sensor data, a first discriminator, the second generative network for inferring semantic labels, and a second discriminator, and

further includes instructions to train the generative neural network using an unsupervised learning process that includes using the first discriminator to assess first output data of the first generative network in comparison with training data and using the second discriminator to assess second output data of the second generative network in comparison with the training data, the training data including training sensor data and training semantic labels associated with at least one training scene.

13. A method, comprising:

in response to receiving a request to generate simulated sensor data for a scene, acquiring simulation data about the scene, wherein the simulation data includes at least simulated information about the scene that is computer-generated;

computing the simulated sensor data using a generative neural network that accepts the simulation data as an input and produces the simulated sensor data as an output, wherein the generative neural network includes a first generative network for producing the simulated sensor data, and a second generative network for inferring semantic labels, wherein the simulated sensor data is a simulated perception of the scene by a sensor; and

training the generative neural network by computing at least a cycle consistency loss that indicates a transitive relationship between learned mappings of the first generative network and the second generative network and that includes a forward cycle consistency and a backward cycle consistency.

14. The method of claim 13 , wherein the simulated data about the scene includes semantic labels identifying objects depicted in the scene and ranges of the objects from a perspective of the sensor,

wherein the simulation data simulates at least range information associated with the scene as perceived by at least one of: an image sensor and a light detection and ranging (LiDAR) sensor, and

wherein the simulated sensor data is intensity data that indicates intensities of reflected light from observed points on the objects, and wherein the intensities correspond with one or more attributes of the objects.

15. The method of claim 13 , wherein acquiring the simulation data includes simulating the scene according to a rendering engine that produces the scene as a three-dimensional environment including objects and spatial relationships between the objects to produce the simulation data, and

wherein the simulation data includes semantic labels and range data for the scene.

16. The method of claim 13 , wherein providing the simulated sensor data includes generating an intensity map and integrating the intensity map with a simulation for the scene to correlate the simulated sensor data with objects in the scene, and

wherein the simulation including the scene is computer-generated and provides inputs for training one or more machine learning tasks.

17. The method of claim 13 , wherein the generative neural network is a generative adversarial network (GAN) that includes the first generative network for producing the simulated sensor data, a first discriminator, the second generative network for inferring semantic labels, and a second discriminator, and

wherein computing the simulated sensor data using the generative neural network includes using learned correlations embodied within the first generative network to infer the simulated sensor data from the simulation data.

18. The method of claim 17 ,

wherein training the generative neural network using an unsupervised learning process includes using the first discriminator to assess first output data of the first generative network in comparison with training data and using the second discriminator to assess second output data of the second generative network in comparison with the training data, the training data including training sensor data and training semantic labels associated with at least one training scene.

19. The method of claim 18 , wherein training includes iteratively computing a loss according to a loss function and adjusting parameters of the generative neural network according to the loss, and

wherein computing the loss includes computing an adversarial loss and a cycle consistency loss, the adversarial loss indicating at least an objective loss associated with the first discriminator identifying whether the first output data is computer generated.