LIDAR-Camera System

1 . Method of spatially calibrating a Light Detection and Ranging, LIDAR, device with respect to at least one camera device, comprising the steps of

capturing by the at least one camera device at least one image of an environment of the at least one camera device;

obtaining a point cloud for the environment by the LIDAR device;

inputting data based on the at least one captured image into a neural network ( 230 , 340 );

outputting by the neural network a neural network representation of the environment of the at least one camera based on the input data;

obtaining a first simulated LIDAR point cloud based on the neural network representation; and

calibrating the LIDAR device by matching of the point cloud and the first simulated LIDAR point cloud.

2 . The method according to claim 1 , wherein the neural network comprises a Multilayer Perceptron, MLP, and further comprising training the MLP to output spatially-dependent volumetric density values for the environment.

3 . The method according to claim 2 , wherein the MLP is trained based on the Neural Radiance Field technique.

4 . The method according to claim 2 , further comprising determining for each of virtual LIDAR rays the accumulated transmittance along the virtual LIDAR ray based on the spatially-dependent volumetric density values, and wherein the first simulated LIDAR point cloud is obtained based on the determined accumulated transmittances.

5 . The method according to claim 1 , further comprising

estimating a rotation and translation of the LIDAR device with respect to the at least one camera device before the obtaining the first simulated LIDAR point cloud; and wherein

the first simulated LIDAR point cloud is obtained using the estimated rotation and estimated translation of the LIDAR device with respect to the at least one camera device; and

the calibrating of the LIDAR device comprises

a) obtaining a first corrected rotation and a first corrected translation of the LIDAR device with respect to the at least one camera device based on the matching of the point cloud and the first simulated LIDAR point cloud;

b) obtaining a second simulated LIDAR point cloud different from the first simulated LIDAR point cloud using the first corrected rotation and the first corrected translation of the LIDAR device with respect to the at least one camera device; and

c) matching the point cloud and the second simulated LIDAR point cloud with each other; and

d) obtaining a more accurate second corrected rotation and a more accurate second corrected translation of the LIDAR device with respect to the at least one camera device based on the matching.

6 . The method according to claim 5 , wherein the steps a) and c) are performed by employing an Iterative Closest Point Algorithm.

7 . The method according to claim 6 , wherein the Iterative Closest Point Algorithm is the Scale-Adaptive Iterative Closest Point Algorithm.

8 . The method according to claim 1 , comprising capturing a plurality of first images of the environment of the at least one camera devices by one of the at least one camera devices;

capturing a plurality of second images of the environment of the at least one camera device by another one of the at least one camera devices; and

inputting data based on the plurality of first captured image and the plurality of second captured images into the neural network;

and wherein

the neural network representation is obtained by the neural network based on the input data based on the plurality of first captured image and the plurality of second captured images.

9 . The method according to claim 1 , wherein the LIDAR device and the at least one camera device are installed on a vehicle.

10 . The method according to claim 9 , wherein the method is performed during movement of the vehicle.

11 . A computer program product comprising computer readable instructions for, when run on a computer, performing the steps of the method according to claim 1 .

12 . Light Detection and Ranging, LIDAR,—camera system, comprising

at least one camera device configured to capture at least one image of an environment of the at least one camera device;

a LIDAR device configured to obtain a point cloud for the environment;

a neural network configured to obtain a neural network representation of the environment of the at least one camera device based on input data provided based on the at least one captured image; and

a processing unit configured to

obtain a first simulated LIDAR point cloud based on the neural network representation; and

calibrate the LIDAR device by matching of the point cloud and the first simulated LIDAR point cloud.

13 . The LIDAR-camera system according to claim 12 , wherein the neural network comprises a Multilayer Perceptron, MLP.

14 . The LIDAR-camera system according to claim 13 , wherein the MLP is trained to output spatially-dependent volumetric density values for the environment.

15 . The LIDAR-camera system according to claim 14 , wherein the MLP is trained based on the Neural Radiance Field technique.

16 . The LIDAR-camera system according to claim 12 , wherein the processing unit is further configured to

estimate a rotation and translation of the LIDAR device with respect to the at least one camera device before the obtaining of the first simulated LIDAR point cloud;

obtain the first simulated LIDAR point cloud based on the estimated rotation and translation of the LIDAR device with respect to the at least one camera device; and

calibrate the LIDAR device by

a) obtaining a first corrected rotation and a first corrected translation of the LIDAR device with respect to the at least one camera device based on the matching of the point cloud and the first simulated LIDAR point cloud;

b) obtaining a second simulated LIDAR point cloud based on the first corrected rotation and first corrected translation of the LIDAR device with respect to the at least one camera device;

c) matching the point cloud and the second simulated LIDAR point cloud with each other; and

d) obtaining a more accurate second corrected rotation and a more accurate second corrected translation of the LIDAR device with respect to the at least one camera device based on the matching.

17 . Vehicle comprising the LIDAR-camera system according to claim 12 .