Tuning simulated data for optimized neural network activation
Techniques described herein are directed to comparing, using a machine-trained model, neural network activations associated with data representing a simulated environment and activations associated with data representing real environment to determine whether the simulated environment is causes similar responses by the neural network, e.g., a detector. If the simulated environment and the real environment do not activate the same way (e.g., the variation between neural network activations of real and simulated data meets or exceeds a threshold), techniques described herein are directed to modifying parameters of the simulated environment to generate a modified simulated environment that more closely resembles the real environment.
1. A method comprising:
receiving first data associated with a real environment;
inputting the first data into a neural network comprising a neural network layer;
receiving, as a first output, a first activation associated with the neural network layer;
receiving second data associated with a simulated environment associated with the real environment;
inputting the second data into the neural network;
receiving, as a second output, a second activation associated with the neural network layer;
based at least in part on the first output and the second output, modifying a parameter associated with a sensor of the simulated environment to generate a modified simulated environment; and
controlling an autonomous vehicle in the real environment based at least in part on a model.
2. The method as claim 1 recites, wherein the first data comprises image data, the second data comprises simulated image data from the simulated environment, and the neural network is associated with a detector.
3. The method as claim 1 recites, wherein the parameter is associated with at least one of brightness, exposure, reflectiveness, light source, or level of light.
4. The method as claim 1 recites, further comprising:
determining, based at least in part on the first output and the second output, a similarity score; and
training the model based at least in part on the similarity score.
5. A system comprising:
one or more processors;
one or more non-transitory computer-readable media that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving first data associated with a real environment;
receiving second data associated with a simulated environment;
inputting the first data into a neural network comprising a layer;
receiving, as a first output, a first activation associated with the layer;
inputting the second data into the neural network;
receiving, as a second output, a second activation associated with the layer;
determining, based at least in part on the first output and the second output, a similarity score representing a similarity between the simulated environment and the real environment;
based at least in part on the similarity score, modifying a parameter associated with a sensor of the simulated environment to generate a modified simulated environment; and
controlling an autonomous vehicle in the real environment based at least in part on a model.
6. The system as claim 5 recites, the operations further comprising:
receiving third data and fourth data associated with the real environment;
inputting the third data into the neural network;
receiving, as a third output, a third activation associated with an additional layer of the neural network;
inputting the fourth data into the neural network;
receiving, as a fourth output, a fourth activation associated with the additional layer;
determining, based at least in part on the third output and the fourth output, a second similarity score; and
training the model based at least in part on the second similarity score.
7. The system as claim 6 recites, wherein the third data and the fourth data are associated with a same portion of the real environment.
8. The system as claim 6 recites, wherein the third data and the fourth data are associated with a different portion of the real environment.
9. The system as claim 5 recites, the operations further comprising:
one or more of testing or validating an algorithm to be used onboard the autonomous vehicle using the modified simulated environment; and
transmitting the algorithm to the autonomous vehicle in the real environment based at least in part on the one or more of the testing or the validating.
10. The system as claim 6 recites, wherein:
the neural network comprises a first layer and a second layer different from the first layer,
the first activation is associated with the first layer,
the second activation is associated with the second layer, and
the similarity score represents a similarity between the first activation of the first layer and the second activation of the second layer.
11. The system as claim 5 recites, the operations further comprising:
associating the first data with a first discretized region;
associating the second data with a second discretized region;
determining a first histogram of activations based at least in part on the first discretized region;
determining a second histogram of activations based at least in part on the second discretized region; and
comparing the first histogram and the second histogram to determine the similarity score.
12. One or more non-transitory computer-readable media that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving first data associated with a real environment;
receiving second data associated with a simulated environment;
inputting the first data into a neural network comprising a first layer and a second layer;
receiving, as a first output, a first activation associated with the first layer;
inputting the second data into the neural network;
receiving, as a second output, a second activation associated with the second layer;
determining, based at least in part on the first output and the second output, a similarity score representing a similarity between the simulated environment and the real environment;
based at least in part on the similarity score, modifying a parameter associated with a sensor of the simulated environment to generate a modified simulated environment; and
controlling an autonomous vehicle in the real environment based at least in part on a model.
13. The one or more non-transitory computer-readable media as claim 12 recites, the operations further comprising:
receiving third data and fourth data associated with the real environment;
inputting the third data into the neural network;
receiving, as a third output, a third activation associated with a third layer of the neural network;
inputting the fourth data into the neural network;
receiving, as a fourth output, a fourth activation associated with the third layer;
determining, based at least in part on the third output and the fourth output, a second similarity score; and
training the model based at least in part on the second similarity score.
14. The one or more non-transitory computer-readable media as claim 13 recites, wherein the third data and the fourth data are associated with:
a same portion of the real environment;
a similar portion of the real environment; and
a different portion of the real environment.
15. The one or more non-transitory computer-readable media as claim 12 recites, wherein the similarity score represents a similarity between the first activation of the first layer and the second activation of the second layer.
16. The one or more non-transitory computer-readable media as claim 12 recites, the operations further comprising:
discretizing the first data and the second data;
determining a first histogram of activations based at least in part on the first data;
determining a second histogram of activations based at least in part on the second data; and
comparing, using the model, the first histogram and the second histogram to determine the similarity score.
17. The one or more non-transitory computer-readable media as claim 12 recites, wherein the first data and the second data comprises image data or LIDAR data,
wherein the model is a first model, and
wherein modifying the parameter comprises:
inputting the similarity score into a second model; and
receiving, from the second model, a revised set of parameters.
18. The method as claim 1 recites, wherein the sensor comprises one of: a light detection and ranging sensor, a radio detection and ranging sensor, an ultrasonic transducer, a sound navigation and ranging sensor, a location sensor, an inertial sensor, an inertial measurement unit, an accelerometer, a magnetometer, a gyroscope, a camera, a wheel encoder, a microphone, an environment sensors, or a Time of Flight sensor.
19. The system as claim 5 recites, wherein the sensor comprises one of: a light detection and ranging sensor, a radio detection and ranging sensor, an ultrasonic transducer, a sound navigation and ranging sensor, a location sensor, an inertial sensor, an inertial measurement unit, an accelerometer, a magnetometer, a gyroscope, a camera, a wheel encoder, a microphone, an environment sensors, or a Time of Flight sensor.
20. The one or more non-transitory computer-readable media as claim 12 recites, wherein the sensor comprises one of: a light detection and ranging sensor, a radio detection and ranging sensor, an ultrasonic transducer, a sound navigation and ranging sensor, a location sensor, an inertial sensor, an inertial measurement unit, an accelerometer, a magnetometer, a gyroscope, a camera, a wheel encoder, a microphone, an environment sensors, or a Time of Flight sensor.