Lane detection method and system based on vision and lidar multi-level fusion

1. A lane detection method based on vision and lidar multi-level fusion, comprising:

calibrating point cloud data and a video image;

constructing a point cloud clustering model by fusing height information, reflection intensity information of the point cloud data, and red, green, blue (RGB) information of the video image, obtaining point clouds of a road based on the point cloud clustering model, and obtaining a lane surface as a first lane candidate region by performing least square fitting on the point clouds of the road;

obtaining four-channel road information by fusing the reflection intensity information of the point cloud data and the RGB information of the video image, inputting the four-channel road information into a trained semantic segmentation network 3D-LaneNet, and outputting an image of a second lane candidate region; and

fusing the first lane candidate region and the second lane candidate region, and combining the first lane candidate region and the second lane candidate region into a final lane region;

wherein the lane detection method is implemented by mounting a lidar and a vehicle-mounted camera on a vehicle.

2. The lane detection method according to claim 1 , wherein the step of constructing the point cloud clustering model by fusing the height information, the reflection intensity information of the point cloud data, and the RGB information of the video image, obtaining the point clouds of the road based on the point cloud clustering model, and obtaining the lane surface as the first lane candidate region by performing least square fitting on the point clouds of the road further comprises:

constructing the point cloud clustering model based on a constraint:

E i =α( H i −H i+1 )+β( Q i −Q i+1 )+γ[( R i −R i+1 )+( G i −G i+1 )+( B i −B i+1 )],

wherein, E i represents a similarity between an i th point and an (i+1) th point; α, β, and γ are weight coefficients; H i is a height of the i th point in the calibrated point cloud data, and Q i is a reflection intensity of the i th point in the calibrated point cloud data; and R i , G i , and B i are RGB three-channel values of an i th pixel in the video image, respectively;

starting clustering by taking a point cloud closest to a central position of a head of the vehicle as a center point and using the point cloud clustering model, wherein when no new point is clustered or after all points in the point cloud are traversed, all point clouds obtained by final clustering are the point clouds of the road; and

performing surface fitting on the point clouds of the road by using a least square method to obtain the lane surface as the first lane candidate region.

3. The lane detection method according to claim 2 , wherein the trained semantic segmentation network 3D-LaneNet processes continuous multi-frame information simultaneously and extracts correlation features of the lane from the continuous multi-frame information; wherein the trained semantic segmentation network 3D-LaneNet comprises twelve 3D-P-Inception modules, wherein a first six 3D-P-Inception modules of the twelve 3D-P-Inception modules are configured for an encode stage, and a second six 3D-P-Inception modules of the twelve 3D-P-Inception modules are configured for a decode stage; wherein the twelve 3D-P-inception modules are obtained by replacing a two-dimensional convolution kernel in Inception-V2 modules with a three-dimensional convolution kernel, and 3D-maxpooling in the twelve 3D-P-Inception modules is replaced with 3D-AvgPooling; wherein convolution kernels of different sizes are used in the twelve 3D-P-Inception modules to facilitate extraction of multi-scale features of the lane.

4. The lane detection method according to claim 3 , further comprising:

training a semantic segmentation network 3D-LaneNet to obtain the trained semantic segmentation network 3D-LaneNet;

wherein the step of training the semantic segmentation network 3D-LaneNet further comprises:

creating a dataset as a training set using calibrated continuous multi-frame point clouds and video images;

when consecutive ten frames of data are input, setting ten initial learning rates a j0 =0.001 for the ten frames of data, respectively, j=1, 2, 3, . . . 10; setting a batch value used for each parameter updating as b=2, and setting a number of times of iterative training as c=5000; calculating a loss function value L j for each frame of fused data by using a cross entropy loss function, and determining a total loss function value

L

=

∑

j

=

1

10

⁢

⁢

L

j

;

initializing a weight W of the semantic segmentation network 3D-LaneNet by using Gaussian weight to follow a distribution X−N(0, σ 2 ), wherein σ 2 is 1.0, 1.1, . . . , 1.9 for a weight of each frame of data, respectively; and initializing a bias of the each frame of data to 0;

inputting samples of the training set into the semantic segmentation network 3D-LaneNet, and after completing an n th iterative training, calculating a ratio

α

n

=

L

jn

/

L

n

of each loss function to a total loss function, wherein L jn is a j th loss function in the n th iterative training, and L n is the total loss function; when α n >0.3 updating a learning rate to a jn =a j0 *(1+α n ) and when α n <0.03, updating the learning rate to a jn =a j0 *(1−10α n ); and

after convergence of network parameters, obtaining the trained semantic segmentation network 3D-LaneNet.

5. The lane detection method according to claim 4 , wherein the step of obtaining the four-channel road information by fusing the reflection intensity information of the point cloud data and the RGB information of the video image, inputting the four-channel road information into the trained semantic segmentation network 3D-LaneNet, and outputting the image of the second lane candidate region further comprises:

representing the RGB information of the i th pixel of the video image by (R i ,G i ,B i ), and performing data standardization on the RGB information (R i ,G i ,B i ) of the i th pixel of the video image by using a Min-Max standardization method to obtain standardized RGB information (R i ′,G i ′,B i ′);

performing data standardization on the reflection intensity Q i of the i th point of the point cloud data by using a z-score standardization method to obtain standardized reflection intensity Q i ′; and

fusing the standardized reflection intensity Q i ′ as fourth-channel information and the standardized RGB information (R i ′,G i ′,B i ′) as three-channel information to obtain the four-channel road information (R i ′, G i ′, B i ′, Q i ′).

6. A lane detection system based on vision and lidar multi-level fusion, comprising a lidar, a vehicle-mounted camera, and a lane detection module, wherein the lane detection module comprises a semantic segmentation network 3D-LaneNet, a calibration unit, a first lane candidate region detection unit, a second lane candidate region detection unit, and a lane fusion unit; wherein

the lidar is configured to obtain point cloud data;

the vehicle-mounted camera is configured to obtain a video image;

the calibration unit is configured to calibrate the point cloud data and the video image;

the first lane candidate region detection unit is configured to construct a point cloud clustering model by fusing height information, reflection intensity information of the point cloud data, and RGB information of the video image, obtain point clouds of a road based on the point cloud clustering model, and obtain a lane surface as a first lane candidate region by performing least square fitting on the point clouds of the road;

the second lane candidate region detection unit is configured to obtain four-channel road information by fusing the reflection intensity information of the point cloud data and the RGB information of the video image, input the four-channel road information into the semantic segmentation network 3D-LaneNet, and output an image of a second lane candidate region; and

the lane fusion unit is configured to fuse the first lane candidate region and the second lane candidate region, and combine the first lane candidate region and the second lane candidate region into a final lane region.

7. The lane detection system according to claim 6 , wherein the first lane candidate region detection unit is implemented as follows:

the point cloud clustering model is constructed based on a constraint:

E i =α( H i −H i+1 )+β( Q i −Q i+1 )+γ[( R i −R i+1 )+( G i −G i+1 )+( B i −B i+1 )],

wherein, E i represents a similarity between an i th point and an (i+1) th point; α, β, and γ are weight coefficients; H i is a height of the i th point in the calibrated point cloud data, and Q i is a reflection intensity of the i th point in the calibrated point cloud data; and R i , G i , and B i are RGB three-channel values of an i th pixel in the video image, respectively;

clustering starts by taking a point cloud closest to a central position of a head of the vehicle as a center point and using the point cloud clustering model, wherein when no new point is clustered or after all points in the point cloud are traversed, all point clouds obtained by final clustering are the point clouds of the road; and

surface fitting is performed on the point clouds of the road by using a least square method to obtain the lane surface as the first lane candidate region.

8. The lane detection system according to claim 7 , wherein the trained semantic segmentation network 3D-LaneNet processes continuous multi-frame information simultaneously and extracts correlation features of the lane from the continuous multi-frame information; wherein the trained semantic segmentation network 3D-LaneNet comprises twelve 3D-P-Inception modules, wherein a first six 3D-P-Inception modules of the twelve 3D-P-Inception modules are configured for an encode stage, and a second six 3D-P-Inception modules of the twelve 3D-P-Inception modules are configured for a decode stage; wherein the twelve 3D-P-inception modules are obtained by replacing a two-dimensional convolution kernel in Inception-V2 modules with a three-dimensional convolution kernel, and 3D-maxpooling in the twelve 3D-P-Inception modules is replaced with 3D-AvgPooling; wherein convolution kernels of different sizes are used in the twelve 3D-P-Inception modules to facilitate extraction of multi-scale features of the lane.

9. The lane detection system according to claim 8 , wherein the semantic segmentation network 3D-LaneNet is trained as follows:

a dataset is created as a training set using calibrated continuous multi-frame point clouds and video images;

when consecutive ten frames of data are input, ten initial learning rates a j0 =0.001 are set for the ten frames of data, respectively, j=1, 2, 3, . . . 10; a batch value used for each parameter updating is set as b=2, and a number of times of iterative training is set as c=5000; a loss function value L j for each frame of fused data is calculated by using a cross entropy loss function, and a total loss function value

L

=

∑

j

=

1

10

⁢

⁢

L

j

is determined; a weight W of the semantic segmentation network 3D-LaneNet is initialized by using Gaussian weight to follow a distribution X−N(0, σ 2 ), wherein σ 2 is 1.0, 1.1, . . . , 1.9 for a weight of each frame of data, respectively; and a bias of the each frame of data is initialized to 0;

samples of the training set are input into the 3D-LaneNet, and after completing an n th iterative training, a ratio

α

n

=

L

jn

/

L

n

of each loss function to a total loss function is calculated, wherein L jn is a j th loss function in the n th iterative training, and L n is the total loss function; when α n >0.3 a learning rate is updated to a jn =a j0 *(1+α n ) and when α n <0.03 the learning rate is updated to α jn =α j0 *(1−10α n ); and

after convergence of network parameters, the trained semantic segmentation network 3D-LaneNet is obtained.

10. The lane detection system according to claim 9 , wherein the second lane candidate region detection unit is implemented as follows:

the RGB information of the i th pixel of the video image is represented by (R i ,G i ,B i ), and data standardization is performed on the RGB information (R i ,G i ,B i ) of the i th pixel of the video image by using a Min-Max standardization method to obtain standardized RGB information (R i ′,G i ′,B i ′);

data standardization is performed on the reflection intensity Q i of the i th point of the point cloud data by using a z-score standardization method to obtain standardized reflection intensity Q i ′; and

the standardized reflection intensity Q as fourth-channel information and the standardized RGB information (R i ′,G i ′,B i ′) as three-channel information are fused to obtain the four-channel road information (R i ′, G i ′, B i ′, Q i ′).