Model training method, image edge detection method, and multi-sensor calibration method
A method for model training, an image edge detection method, a multi-sensor calibration method, a computer program product, and a computer device is disclosed. The model is used to generate occlusion relationships between pixel pairs. The model training method comprises: constructing an initial model; obtaining multiple training images and reference annotation results for each training image, wherein each training image comprises RGB features and depth features, and the reference annotation results for each training image are annotation results of occlusion relationships between adjacent pixel pairs in said training image generated based on the depth features of said training image; respectively using the RGB features of the multiple training images as inputs to the initial model, using the reference annotation results corresponding to the input training images as outputs of the initial model, and training the initial model to obtain a well-trained model.
1 . A calibration method for multiple sensors, comprising:
obtaining image data from a first sensor and 3D point cloud data from a second sensor, wherein the 3D point cloud data is synchronized with an acquisition time of the image data;
extracting a first occlusion edge feature from the image data based on occlusion relationships between adjacent pixel pairs in the image data, wherein the first occlusion edge feature includes a position and orientation of the first occlusion edge;
extracting a second occlusion edge feature from the 3D point cloud data based on occlusion relationships between adjacent point cloud points in the 3D point cloud data, wherein the second occlusion edge feature includes a position and orientation of the second occlusion edge;
matching multiple pixel points included in the first occlusion edge and multiple point cloud points included in the second occlusion edge to obtain multiple successfully matched pairs of points composed of pixel points and point cloud points; and
obtaining a transformation relationship between a coordinate system of the first sensor and the coordinate system of the second sensor based on the multiple pairs of points, for calibrating the first sensor and the second sensor.
2 . The calibration method according to claim 1 , wherein obtaining the 3D point cloud data from the second sensor comprises:
based on an initial calibration of the first sensor and the second sensor, obtaining 3D point cloud data from a single frame acquired by the second sensor that is within a field of view of the first sensor.
3 . The calibration method according to claim 2 further comprising:
extracting the first occlusion edge feature by (i) obtaining an image to be processed, (ii) employing a trained model to generate annotation results for occlusion relationships in adjacent pixel pairs in the image to be processed, and (iii) extracting occlusion edge features from the image to be processed based on the annotation results, wherein the occlusion edge features comprise the positions and orientations of occlusion edges; or
using a method of manual annotation to generate occlusion relationships between adjacent pixel pairs in the image data, and extracting the first occlusion edge feature based on the generated occlusion relationships in the adjacent pixel pairs.
4 . The calibration method according to claim 3 further comprising:
projecting multiple point cloud points included in the second occlusion edge onto the image data to obtain the positions and orientations of corresponding projected occlusion edges; and
matching first occlusion edges with a same orientation as the projected occlusion edges to obtain multiple successfully matched pairs of points.
5 . The calibration method according to claim 3 , wherein:
the annotation results comprise: annotations for occluding pixels and annotations for occluded pixels, and occlusion direction annotations for adjacent pixel pairs with occlusion relationships in the image to be processed, wherein the occlusion direction indicates the direction from the occluding pixel to the occluded pixel; and
extracting occlusion edge features from the image to be processed based on the annotation results comprises: determining the positions of the occlusion edge features based on locations of occluding pixels included in the annotation results, and determining the orientations of the occlusion edge features based on the occlusion direction annotations for adjacent pixel pairs included in the annotation results.
6 . The calibration method according to claim 5 , wherein the occlusion edge features are composed of occluding pixels from multiple sets of adjacent pixel pairs in an occlusion relationship;
when the occlusion directions of the multiple sets of adjacent pixel pairs in an occlusion relationship are horizontally pointing right, the orientation of the occlusion edge feature is a right occluded edge;
when the occlusion directions of the multiple sets of adjacent pixel pairs in an occlusion relationship are horizontally pointing left, the orientation of the occlusion edge feature is a left occluded edge;
when the occlusion directions of the multiple sets of adjacent pixel pairs in an occlusion relationship are vertically pointing upward, the orientation of the occlusion edge feature is an upper occluded edge; and
when the occlusion directions of the multiple sets of adjacent pixel pairs in an occlusion relationship are vertically pointing downward, the orientation of the occlusion edge feature is a lower occluded edge.
7 . The calibration method according to claim 3 , further comprising:
generating annotation results of occlusion relationships of adjacent pixel pairs arranged in different directions in the image to be processed, based on the trained model; and
extracting occlusion edge features with different orientations in the image to be processed based on the annotation results.
8 . The calibration method according to claim 7 , wherein the different directions comprise at least two of a horizontal direction, a vertical direction, a diagonal direction, and an anti-diagonal direction of the image to be processed.
9 . The calibration method according to claim 3 , wherein:
the image to be processed is an RGB image; and/or
the image to be processed is from images captured by an image sensor in traffic road scenes.
10 . The calibration method according to claim 1 , wherein the extracted first occlusion edge feature includes first occlusion edges with various orientations located at multiple positions, and the extracted second occlusion edge feature includes second occlusion edges with various orientations located at multiple positions, the method further comprising:
matching first occlusion edges and second occlusion edges that are positionally adjacent and have a same orientation to obtain multiple successfully matched pairs of points composed of pixel points and point cloud points.
11 . The calibration method according to claim 1 , wherein the orientation of the first occlusion edge is determined based on occlusion directions of adjacent pixel pairs near the first occlusion edge, and the orientation of the second occlusion edge is determined based on occlusion directions of adjacent point cloud points near the second occlusion edge.
12 . The calibration method according to claim 1 , wherein the orientations of the first occlusion edge and the second occlusion edge both include at least one of: a left occlusion edge, a right occlusion edge, an upper occlusion edge, and a lower occlusion edge.
13 . The calibration method according to claim 1 , further comprising: annotating occlusion relationships of adjacent point cloud points based on a difference in depth values of adjacent point cloud points in the 3D point cloud data; and extracting a second occlusion edge feature from the 3D point cloud data according to an occlusion relationship of the annotated adjacent point cloud points.
14 . The calibration method according to claim 1 , characterized in that the calibration method comprises: iteratively performing steps for calibrating the first sensor and the second sensor.
15 . The calibration method according to claim 1 , wherein:
the first sensor comprises an onboard camera or a camera, and the second sensor comprises an onboard lidar; and/or
relative positions of the first sensor and the second sensor are fixed.
16 . A computer program product, characterized in that the computer program product comprises a computer program stored on a non-transitory computer readable medium, which, when executed by a processor, implements a calibration method according to claim 1 .
17 . A computer device comprising a processor, memory, and a computer program stored in the memory, characterized in that when the computer program is executed by the processor, it implements a calibration method according to claim 1 .