Method and system for fusing data from LiDAR and camera
A LiDAR and camera data fusion method includes generating a voxel-wise feature map based on point cloud data of a LiDAR sensor, generating a pixel-wise feature map based on image data of a camera, converting three-dimensional ( 3 D) coordinates of point data of the voxel-wise feature map to two-dimensional ( 2 D) coordinates, based on at least one predefined calibration parameter, and generating fused data by combining pixel data of the pixel-wise feature map and point data of the 2 D coordinates.
1 . A Light Detection and Ranging (LiDAR) and camera data fusion method, comprising:
generating, by a processor, a voxel-wise feature map based on point cloud data of a LiDAR sensor;
generating, by the processor, a pixel-wise feature map based on image data of a camera;
converting, by the processor, three-dimensional (3D) coordinates of point data of the voxel-wise feature map to two-dimensional (2D) coordinates, based on at least one predefined calibration parameter; and
generating, by the processor, fused data by combining pixel data of the pixel-wise feature map and point data of the 2D coordinates.
2 . The LiDAR and camera data fusion method of claim 1 , wherein the voxel-wise feature map is generated via a 3D voxel-based neural network.
3 . The LiDAR and camera data fusion method of claim 1 , wherein the pixel-wise feature map is generated via a neural network.
4 . The LiDAR and camera data fusion method of claim 1 , wherein the at least one predefined calibration parameter includes an angle difference and a separation distance between the LiDAR and the camera on a vehicle.
5 . The LiDAR and camera data fusion method of claim 1 , further including:
determining, by the processor, on the pixel-wise feature map, a pixel corresponding to each of the point data of the 2D coordinates;
sampling, by the processor, in the pixel-wise feature map, pixels in an area of a predetermined size including the pixel corresponding to each of the point data; and
generating, by the processor, a pixel feature vector representative of the sampled pixels through max pooling.
6 . The LiDAR and camera data fusion method of claim 5 , wherein the generating of fused data includes generating a fused feature vector via a convolutional layer, after combining a voxel feature vector of the point data and the pixel feature vector.
7 . The LiDAR and camera data fusion method of claim 6 , further including:
extracting, by the processor, the voxel feature vector of the point data by randomly selecting the point data included in voxels of the voxel-wise feature map.
8 . The LiDAR and camera data fusion method of claim 6 , wherein the generating of the fused feature vector via the convolutional layer includes:
applying a convolution operation to the voxel feature vector of the point data and the fused feature vector;
applying a batch normalization operation to a result of the convolution operation; and
applying a rectified linear unit (ReLU) operation to a result of the batch normalization operation to generate the fused feature vector.
9 . The LiDAR and camera data fusion method of claim 1 , further including updating and outputting the voxel-wise feature map based on the fused data.
10 . A LiDAR and camera data fusion system, comprising:
an interface configured to receive point cloud data from a LiDAR sensor and image data from a camera;
a memory configured to store at least one predefined calibration parameter; and
a processor electrically or communicatively connected to the interface and the memory,
wherein the processor is configured to:
generate a voxel-wise feature map based on the point cloud data;
generate a pixel-wise feature map based on the image data;
convert three-dimensional (3D) coordinates of point data of the voxel-wise feature map to two-dimensional (2D) coordinates, based on the at least one predefined calibration parameter; and
generate fused data by fusing pixel data of the pixel-wise feature map and the point data of the 2D coordinates.
11 . The LiDAR and camera data fusion system of claim 10 , wherein the voxel-wise feature map is generated via a 3D voxel-based neural network.
12 . The LiDAR and camera data fusion system of claim 10 , wherein the pixel-wise feature map is generated via a neural network.
13 . The LiDAR and camera data fusion system of claim 10 , wherein the least one predefined calibration parameter includes an angle difference and a separation distance between the LiDAR and the camera on a vehicle.
14 . The LiDAR and camera data fusion system of claim 10 , wherein the processor is further configured to:
determine, in the pixel-wise feature map, a pixel corresponding to each of the point data of the 2D coordinates;
sample, in the pixel-wise feature map, pixels in an area of a predetermined size including the pixel corresponding to each of the point data; and
generate a pixel feature vector representative of the sampled pixels through max pooling.
15 . The LiDAR and camera data fusion system of claim 14 , wherein the processor is further configured to generate a fused feature vector via a convolutional layer, after combining a voxel feature vector of the point data and the pixel feature vector.
16 . The LiDAR and camera data fusion system of claim 15 , wherein the processor is further configured to extract the voxel feature vector of the point data by randomly selecting point data included in voxels of the voxel-wise feature map.
17 . The LiDAR and camera data fusion system of claim 15 , wherein the processor is further configured to:
apply a convolution operation to the voxel feature vector of the point data and the fused feature vector;
apply a batch normalization operation to a result of the convolution operation; and
apply a rectified linear unit (ReLU) operation to a result of the batch normalization operation to generate the fused feature vector.
18 . The LiDAR and camera data fusion system of claim 10 , wherein the processor is further configured to update and output the voxel-wise feature map based on the fused data.
19 . A vehicle comprising:
a LiDAR and camera data fusion system,
wherein the system includes:
an interface configured to receive point cloud data from a LiDAR sensor and image data from a camera;
a memory configured to store at least one predefined calibration parameter; and
a processor electrically or communicatively connected to the interface and the memory, and
wherein the processor is configured to:
generate a voxel-wise feature map based on the point cloud data;
generate a pixel-wise feature map based on the image data;
convert three-dimensional (3D) coordinates of point data of the voxel-wise feature map to two-dimensional (2D) coordinates, based on the at least one predefined calibration parameter; and
generate fused data by fusing pixel data of the pixel-wise feature map and the point data of the 2D coordinates.