Image data processing method, electronic device, and computer readable storage medium using neural networks and homography matrix

1 . An image data processing method, comprising:

processing a first image and a second image by using a first neural network to obtain a homography matrix, wherein the first image is captured at a first moment, the second image is captured at a second moment, and the first image and the second image have road-surface elements in a same area;

determining a mapped image feature of a first image feature based on the homography matrix, wherein the first image feature is a feature extracted based on the first image;

fusing the mapped image feature and a second image feature to obtain a fused image feature, wherein the second image feature is a feature extracted based on the second image; and

processing the fused image feature by using a second neural network to obtain a first pixel height-to-depth ratio of the second image.

2 . The image data processing method according to claim 1 , wherein the processing the first image and the second image by using the first neural network to obtain the homography matrix comprises:

fusing the first image feature of the first image and the second image feature of the second image to obtain a third image feature;

processing the third image feature by using a road-surface subnetwork in the first neural network to determine road-surface normal information;

processing the third image feature by using a posture subnetwork in the first neural network to determine a relative camera posture between the first image and the second image; and

determining the homography matrix based on the road-surface normal information, the relative camera posture, and a pre-stored height of a camera relative to a road surface.

3 . The image data processing method according to claim 2 , wherein after the processing the first image and the second image by using the first neural network to obtain the homography matrix, the method further comprises:

performing image reconstruction on the first image by using the homography matrix to obtain a first reconstructed image;

adjusting a matrix parameter of the homography matrix based on a pixel displacement of the road-surface elements in the same area, which is between the first reconstructed image and the second image; and

performing parameter adjustment on the road-surface subnetwork and the posture subnetwork based on the homography matrix whose matrix parameter is adjusted.

4 . The image data processing method according to claim 1 , wherein after the processing the fused image feature by using the second neural network to obtain the first pixel height-to-depth ratio of the second image, the method further comprises:

during collection time of the second image, determining a second pixel height-to-depth ratio of a target object in the second image based on radar scanning data corresponding to the target object in the second image; and

performing parameter adjustment on the second neural network based on a difference between the first pixel height-to-depth ratio and the second pixel height-to-depth ratio.

5 . The image data processing method according to claim 1 , wherein after the processing the fused image feature by using the second neural network to obtain the first pixel height-to-depth ratio of the second image, the method further comprises:

during collection time of the second image, determining an overall loss value based on the first pixel height-to-depth ratio, a road-surface mask of the second image, and radar scanning data corresponding to a target object in the second image; and

performing parameter adjustment on the first neural network and the second neural network based on the overall loss value.

6 . The image data processing method according to claim 5 , wherein the during collection time of the second image, determining the overall loss value based on the first pixel height-to-depth ratio, the road-surface mask of the second image, and the radar scanning data corresponding to the target object in the second image comprises:

determining a second pixel height-to-depth ratio of the target object based on the radar scanning data;

determining a first loss value based on the first pixel height-to-depth ratio and the second pixel height-to-depth ratio;

performing image reconstruction on the first image by using the homography matrix to obtain a first reconstructed image;

determining a pixel displacement between a first image area and a second image area based on the first pixel height-to-depth ratio, wherein the first image area is a remaining image area except a road-surface image area in the first reconstructed image, and the second image area is a remaining image area except a road-surface image area in the second image;

adjusting a pixel position of the first reconstructed image based on the pixel displacement to obtain a second reconstructed image;

determining a second loss value based on the second reconstructed image, the second image, and the road-surface mask of the second image; and

determining the overall loss value based on the first loss value and the second loss value.

7 . The image data processing method according to claim 6 , wherein the determining the second loss value based on the second reconstructed image, the second image, and the road-surface mask of the second image comprises:

determining an entire-image photometric error between the second reconstructed image and the second image;

determining photometric errors between the second reconstructed image and the second image in the road-surface image area based on the entire-image photometric error and the road-surface mask of the second image; and

determining the second loss value based on the entire-image photometric error and the photometric errors between the second reconstructed image and the second image in the road-surface image area.

8 . A non-transitory computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is used for implementing an image data processing method,

wherein the image data processing method comprises:

processing a first image and a second image by using a first neural network to obtain a homography matrix, wherein the first image is captured at a first moment, the second image is captured at a second moment, and the first image and the second image have road-surface elements in a same area;

determining a mapped image feature of a first image feature based on the homography matrix, wherein the first image feature is a feature extracted based on the first image;

fusing the mapped image feature and a second image feature to obtain a fused image feature, wherein the second image feature is a feature extracted based on the second image; and

processing the fused image feature by using a second neural network to obtain a first pixel height-to-depth ratio of the second image.

9 . The non-transitory computer readable storage medium according to claim 8 , wherein the processing the first image and the second image by using the first neural network to obtain the homography matrix comprises:

fusing the first image feature of the first image and the second image feature of the second image to obtain a third image feature;

processing the third image feature by using a road-surface subnetwork in the first neural network to determine road-surface normal information;

processing the third image feature by using a posture subnetwork in the first neural network to determine a relative camera posture between the first image and the second image; and

determining the homography matrix based on the road-surface normal information, the relative camera posture, and a pre-stored height of a camera relative to a road surface.

10 . The non-transitory computer readable storage medium according to claim 9 , wherein after the processing the first image and the second image by using the first neural network to obtain the homography matrix, the method further comprises:

performing image reconstruction on the first image by using the homography matrix to obtain a first reconstructed image;

adjusting a matrix parameter of the homography matrix based on a pixel displacement of the road-surface elements in the same area, which is between the first reconstructed image and the second image; and

performing parameter adjustment on the road-surface subnetwork and the posture subnetwork based on the homography matrix whose matrix parameter is adjusted.

11 . The non-transitory computer readable storage medium according to claim 8 , wherein after the processing the fused image feature by using the second neural network to obtain the first pixel height-to-depth ratio of the second image, the method further comprises:

during collection time of the second image, determining a second pixel height-to-depth ratio of a target object in the second image based on radar scanning data corresponding to the target object in the second image; and

performing parameter adjustment on the second neural network based on a difference between the first pixel height-to-depth ratio and the second pixel height-to-depth ratio.

12 . The non-transitory computer readable storage medium according to claim 8 , wherein after the processing the fused image feature by using the second neural network to obtain the first pixel height-to-depth ratio of the second image, the method further comprises:

during collection time of the second image, determining an overall loss value based on the first pixel height-to-depth ratio, a road-surface mask of the second image, and radar scanning data corresponding to a target object in the second image; and

performing parameter adjustment on the first neural network and the second neural network based on the overall loss value.

13 . The non-transitory computer readable storage medium according to claim 12 , wherein the during collection time of the second image, determining the overall loss value based on the first pixel height-to-depth ratio, the road-surface mask of the second image, and the radar scanning data corresponding to the target object in the second image comprises:

determining a second pixel height-to-depth ratio of the target object based on the radar scanning data;

determining a first loss value based on the first pixel height-to-depth ratio and the second pixel height-to-depth ratio;

performing image reconstruction on the first image by using the homography matrix to obtain a first reconstructed image;

determining a pixel displacement between a first image area and a second image area based on the first pixel height-to-depth ratio, wherein the first image area is a remaining image area except a road-surface image area in the first reconstructed image, and the second image area is a remaining image area except a road-surface image area in the second image;

adjusting a pixel position of the first reconstructed image based on the pixel displacement to obtain a second reconstructed image;

determining a second loss value based on the second reconstructed image, the second image, and the road-surface mask of the second image; and

determining the overall loss value based on the first loss value and the second loss value.

14 . The non-transitory computer readable storage medium according to claim 13 , wherein the determining the second loss value based on the second reconstructed image, the second image, and the road-surface mask of the second image comprises:

determining an entire-image photometric error between the second reconstructed image and the second image;

determining photometric errors between the second reconstructed image and the second image in the road-surface image area based on the entire-image photometric error and the road-surface mask of the second image; and

determining the second loss value based on the entire-image photometric error and the photometric errors between the second reconstructed image and the second image in the road-surface image area.

15 . An electronic device, wherein the electronic device comprises:

a processor; and

a memory, configured to store processor-executable instructions,

wherein the processor is configured to read the executable instructions from the memory, and execute the instructions to implement an image data processing method,

wherein the image data processing method comprises:

processing a first image and a second image by using a first neural network to obtain a homography matrix, wherein the first image is captured at a first moment, the second image is captured at a second moment, and the first image and the second image have road-surface elements in a same area;

determining a mapped image feature of a first image feature based on the homography matrix, wherein the first image feature is a feature extracted based on the first image;

fusing the mapped image feature and a second image feature to obtain a fused image feature, wherein the second image feature is a feature extracted based on the second image; and

processing the fused image feature by using a second neural network to obtain a first pixel height-to-depth ratio of the second image.

16 . The electronic device according to claim 15 , wherein the processing the first image and the second image by using the first neural network to obtain the homography matrix comprises:

fusing the first image feature of the first image and the second image feature of the second image to obtain a third image feature;

processing the third image feature by using a road-surface subnetwork in the first neural network to determine road-surface normal information;

processing the third image feature by using a posture subnetwork in the first neural network to determine a relative camera posture between the first image and the second image; and

determining the homography matrix based on the road-surface normal information, the relative camera posture, and a pre-stored height of a camera relative to a road surface.

17 . The electronic device according to claim 16 , wherein after the processing the first image and the second image by using the first neural network to obtain the homography matrix, the method further comprises:

performing image reconstruction on the first image by using the homography matrix to obtain a first reconstructed image;

adjusting a matrix parameter of the homography matrix based on a pixel displacement of the road-surface elements in the same area, which is between the first reconstructed image and the second image; and

performing parameter adjustment on the road-surface subnetwork and the posture subnetwork based on the homography matrix whose matrix parameter is adjusted.

18 . The electronic device according to claim 15 , wherein after the processing the fused image feature by using the second neural network to obtain the first pixel height-to-depth ratio of the second image, the method further comprises:

during collection time of the second image, determining a second pixel height-to-depth ratio of a target object in the second image based on radar scanning data corresponding to the target object in the second image; and

performing parameter adjustment on the second neural network based on a difference between the first pixel height-to-depth ratio and the second pixel height-to-depth ratio.

19 . The electronic device according to claim 15 , wherein after the processing the fused image feature by using the second neural network to obtain the first pixel height-to-depth ratio of the second image further, the method further comprises:

during collection time of the second image, determining an overall loss value based on the first pixel height-to-depth ratio, a road-surface mask of the second image, and radar scanning data corresponding to a target object in the second image; and

performing parameter adjustment on the first neural network and the second neural network based on the overall loss value.

20 . The electronic device according to claim 19 , wherein the during collection time of the second image, determining the overall loss value based on the first pixel height-to-depth ratio, the road-surface mask of the second image, and the radar scanning data corresponding to the target object in the second image comprises:

determining a second pixel height-to-depth ratio of the target object based on the radar scanning data;

determining a first loss value based on the first pixel height-to-depth ratio and the second pixel height-to-depth ratio;

performing image reconstruction on the first image by using the homography matrix to obtain a first reconstructed image;

determining a pixel displacement between a first image area and a second image area based on the first pixel height-to-depth ratio, wherein the first image area is a remaining image area except a road-surface image area in the first reconstructed image, and the second image area is a remaining image area except a road-surface image area in the second image;

adjusting a pixel position of the first reconstructed image based on the pixel displacement to obtain a second reconstructed image;

determining a second loss value based on the second reconstructed image, the second image, and the road-surface mask of the second image; and

determining the overall loss value based on the first loss value and the second loss value.