MULTI-CHANNEL SENSOR SIMULATION FOR AUTONOMOUS CONTROL SYSTEMS
An autonomous control system combines sensor data from multiple sensors to simulate sensor data from high-capacity sensors. The sensor data contains information related to physical environments surrounding vehicles for autonomous guidance. For example, the sensor data may be in the form of images that visually capture scenes of the surrounding environment, geo-location of the vehicles, and the like. The autonomous control system simulates high-capacity sensor data of the physical environment from replacement sensors that may each have lower capacity than high-capacity sensors. The high-capacity sensor data may be simulated via one or more neural network models. The autonomous control system performs various detection and control algorithms on the simulated sensor data to guide the vehicle autonomously.
1 . (canceled)
2 . A method of controlling a vehicle with objects identified in an environment with simulated sensor data, comprising:
obtaining sensor data from a plurality of low-capacity sensors positioned on a vehicle in a real-world environment, wherein the low-capacity sensors are passive image sensors;
applying a neural network model having a single-branch input structure to a synthesized input of the obtained sensor data, wherein the synthesized input is input into a single layer of the network model, wherein the neural network model determines simulated information associated with high-capacity sensors; and
adjusting a control parameter of the vehicle based on one or more objects identified, at least in part, using the simulated information.
3 . The method of claim 2 , wherein the obtained data is arranged in one or more channels of an input data structure.
4 . The method of claim 3 , wherein each channel corresponds to a respective one of the plurality of low-capacity sensors.
5 . The method of claim 2 , wherein the obtained data is concatenated in the horizontal and/or vertical dimension to form a panorama that combines images with an overlapping field-of-view into a single image.
6 . The method of claim 2 , wherein content-aware panorama stitching techniques are applied to the obtained data.
7 . The method of claim 2 , wherein sensor-specific reshaping of the obtained sensor data is performed prior to inputting the synthesized input into the neural network model.
8 . A method of controlling a vehicle with objects identified in an environment with simulated sensor data, comprising:
obtaining sensor data from a plurality of low-capacity sensors positioned on a vehicle in a real-world environment, wherein the low-capacity sensors are passive image sensors;
applying a neural network model having a multiple-branch input structure to the obtained sensor data, wherein different portions of the obtained sensor data are input into corresponding branches of the neural network model, wherein the neural network model determines simulated information associated with high-capacity sensors; and
adjusting a control parameter of the vehicle based on one or more objects identified, at least in part, using the simulated information.
9 . The method of claim 8 , wherein each of the different portions of the obtained sensor data corresponds to a unique one of the plurality of low-capacity sensors, and wherein each branch comprises one or more neural network layers configured to learn behavior unique to the corresponding low-capacity sensor.
10 . The method of claim 8 , wherein one of the different portions of the obtained sensor data comprises RGB camera images and another one of the different portions of the obtained sensor data comprises LIDAR sensor data.
11 . The method of claim 8 , wherein reshaping of each portion of the obtained sensor data is performed within the corresponding branch of the neural network model.
12 . The method of claim 11 , wherein reshaping is performed using interpolation techniques.
13 . The method of claim 11 , wherein reshaping is performed using a convolution or deconvolution layer having a non-unit stride.
14 . The method of claim 8 , wherein each of the portions of the obtained sensor data corresponds to different timesteps.
15 . The method of claim 8 , wherein the branches of the neural network model are merged partway through the neural network model.
16 . A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to:
obtain sensor data from a plurality of low-capacity sensors positioned on a vehicle in a real-world environment, wherein the low-capacity sensors are passive image sensors;
apply a neural network model having a single-branch input structure to a synthesized input of the obtained sensor data, wherein the synthesized input is input into a single layer of the network model, wherein the neural network model determines simulated information associated with high-capacity sensors; and
adjust a control parameter of the vehicle based on one or more objects identified, at least in part, using the simulated information.
17 . The system of claim 16 , wherein the obtained data is arranged in one or more channels of an input data structure.
18 . The system of claim 17 , wherein each channel corresponds to a respective one of the plurality of low-capacity sensors.
19 . The system of claim 16 , wherein the obtained data is concatenated in the horizontal and/or vertical dimension to form a panorama that combines images with an overlapping field-of-view into a single image.
20 . The system of claim 16 , wherein content-aware panorama stitching techniques are applied to the obtained data.
21 . The system of claim 16 , wherein sensor-specific reshaping of the obtained sensor data is performed prior to inputting the synthesized input into the neural network model.