Simulated LiDAR devices and systems
Systems and methods for generating simulated LiDAR data using RADAR and image data are provided. An algorithm is trained using deep-learning techniques such as loss functions to generate simulated LiDAR data using RADAR and image data. Once trained, the algorithm can be implemented in a system, such as a vehicle, equipped with RADAR and image sensors in order to generate simulated LiDAR data describing the system's environment. The simulated LiDAR data may be used by a vehicle control system to determine, generate, and implement modified driving operations.
1. A system comprising:
a first sensor configured to sense data signals;
a second sensor configured to sense image signals;
an encoder, coupled to said first sensor and said second sensor, to receive at least one of said data signals and at least one of said image signals, said encoder comprising a convolutional neural network autoencoder;
a rendering engine coupled to said encoder;
a processor coupled to said encoder and to said rendering engine; and
a computer readable medium coupled to said processor and comprising instructions stored thereon that cause the processor to perform at least the following operations:
applying, by said encoder, convolutional neural network autoencoding operations;
outputting, by said encoder, at least one of a predicted point cloud and a predicted depth map based on said at least one of said data signals and said at least one of said image signals;
rendering, by said rendering engine, said at least one of a predicted point cloud and a predicted depth map into at least one of a second point cloud and a second depth map of lower density than said predicted point cloud; and
generating, by said rendering engine, based on said at least one of a second point cloud and a second depth map, a simulated LiDAR output capable of being used in place of a non-simulated LiDAR output.
2. The system of claim 1 , wherein said first sensor comprises a RADAR array, and said data signals comprise RADAR signals.
3. The system of claim 1 , wherein said second sensor comprises a camera.
4. The system of claim 1 , wherein said at least one of said data signals consists of a RADAR point cloud.
5. The system of claim 1 , wherein said at least one of said image signals consists of a single frame.
6. The system of claim 1 , wherein said simulated LiDAR output comprises a simulated LiDAR point cloud.
7. The system of claim 1 , wherein said simulated LiDAR output comprises a depth map.
8. A vehicle comprising:
a simulated LiDAR imaging system comprising:
a first sensor configured to sense data signals;
a second sensor configured to sense image signals;
an encoder, coupled to said first sensor and to said second sensor, configured to receive at least one of said data signals and at least one of said image signals, said encoder comprising a convolutional neural network autoencoder;
a rendering engine coupled to said encoder;
a first processor coupled to said encoder and to said rendering engine; and
a first computer readable medium coupled to said first processor and comprising instructions stored thereon that cause said first processor to perform at least the following operations:
applying, by said encoder, convolutional neural network autoencoding operations;
outputting, by said encoder, at least one of a predicted point cloud and a predicted depth map based on said at least one of said data signals and said at least one of said image signals;
rendering, by said rendering engine, said at least one of a predicted point cloud and a predicted depth map into at least one of a second point cloud and a second depth map of lower density than said predicted point cloud; and
generating, by said rendering engine, based on said at least one of a second point cloud and a second depth map, a simulated LiDAR output capable of being used in place of a non-simulated LiDAR output;
a second processor that receives said simulated LiDAR output; and
a second computer readable medium coupled to said second processor and comprising instructions stored thereon that cause said second processor to:
generate at least one modified driving operation signal in response to said simulated LiDAR output; and
communicate said modified driving operation signal to one or more actuators to control one or more vehicle components to perform a selected operation in response to said modified driving operation signal.
9. The vehicle of claim 8 , wherein said first sensor comprises a RADAR array, and said data signals comprise RADAR signals.
10. The vehicle of claim 8 , wherein said second sensor comprises a camera.
11. The vehicle of claim 8 , wherein said at least one of said data signals consists of a RADAR point cloud.
12. The vehicle of claim 8 , wherein said at least one of said image signals consists of a single frame.
13. The vehicle of claim 8 , wherein said simulated LiDAR output comprises a simulated LiDAR point cloud.
14. The vehicle of claim 8 , wherein said simulated LiDAR output comprises a depth map.
15. A method comprising:
sensing, by a first sensor, at least one data signal;
sensing, by a second sensor, at least one image signal;
receiving, by an encoder coupled to said first sensor and said second sensor, said at least one data signal and said at least one image signal, said encoder comprising a convolutional neural network autoencoder;
applying, by said encoder, convolutional neural network autoencoding operations;
outputting, by said encoder, at least one of a predicted point cloud and a predicted depth map based on said at least one of said data signal and said at least one of said image signal;
rendering, by a rendering engine, said at least one of a predicted point cloud and a predicted depth map into at least one of a second point cloud and a second depth map of lower density than said predicted point cloud; and
generating, by said rendering engine, based on said at least one of a second point cloud and a second depth map, a simulated LiDAR output capable of being used in place of a non-simulated LiDAR output.
16. The method of claim 15 , wherein said first sensor comprises a RADAR array, and said at least one data signal comprise RADAR signals.
17. The method of claim 15 , wherein said at least one data signal consists of a RADAR point cloud.
18. The method of claim 15 , wherein said at least one image signal consists of a single frame.
19. The method of claim 15 , wherein said simulated LiDAR output comprises a simulated LiDAR point cloud.
20. The method of claim 15 , wherein said simulated LiDAR output comprises a depth map.