Method of processing image, method of training model, and electronic device
A method of processing an image, a method of training a multi-task processing model, and an electronic device, which relate to a field of an automatic driving technology, in particular to a field of high-definition map technology. The method of processing an image includes: processing a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image, and a dense descriptor map of the to-be-processed image; determining a pair of matched feature points between the to-be-processed image and a reference image based on the feature point and the feature point descriptor map; and determining a pair of matched pixels between the to-be-processed image and the reference image based on the dense descriptor map.
1. A method of processing an image, comprising:
processing a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image, and a dense descriptor map of the to-be-processed image;
determining a pair of matched feature points between the to-be-processed image and a reference image, based on the feature point and the feature point descriptor map; and
determining a pair of matched pixels between the to-be-processed image and the reference image, based on the dense descriptor map,
wherein the processing a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image and a dense descriptor map of the to-be-processed image comprises:
inputting the to-be-processed image into a multi-task processing model to obtain the feature point of the to-be-processed image, the feature point descriptor map of the to-be-processed image and the dense descriptor map of the to-be-processed image,
wherein the multi-task processing model comprises a feature point classification branch, and the method further comprises:
inputting the to-be-processed image into the feature point classification branch to obtain a feature point category result, so as to determine the pair of matched feature points between the to-be-processed image and the reference image based on the feature point, the feature point descriptor map and the feature point category result,
wherein the determining a pair of matched feature points between the to-be-processed image and a reference image based on the feature point and the feature point descriptor map comprises:
screening the feature point based on the feature point category result, so as to determine a target feature point; and
determining the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor map, and
wherein the determining the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor map comprises:
extracting, from the feature point descriptor map, a feature point descriptor matched with the target feature point based on the target feature point; and
determining, by using a feature point matching method, the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor matched with the target feature point.
2. The method according to claim 1 , wherein the multi-task processing model further comprises a feature point extraction branch, a feature point descriptor map calculation branch and a dense descriptor map calculation branch, and
the inputting the to-be-processed image into a multi-task processing model to obtain the feature point of the to-be-processed image, the feature point descriptor map of the to-be-processed image and the dense descriptor map of the to-be-processed image comprises:
inputting the to-be-processed image into the feature point extraction branch to obtain the feature point;
inputting the to-be-processed image into the feature point descriptor map calculation branch to obtain the feature point descriptor map; and
inputting the to-be-processed image into the dense descriptor map calculation branch to obtain the dense descriptor map.
3. The method according to claim 2 , wherein the feature point extraction branch comprises a plurality of feature point extraction sub branches, and
the inputting the to-be-processed image into the feature point extraction branch to obtain the feature point comprises:
inputting the to-be-processed image into the plurality of feature point extraction sub branches, so as to obtain a plurality of feature point scoring maps, wherein the plurality of feature point scoring maps correspond to the plurality of feature point extraction sub branches one by one;
performing a weighted summation on the plurality of feature point scoring maps to obtain a target feature point scoring map; and
performing an interpolation processing on the target feature point scoring map to obtain the feature point.
4. The method according to claim 2 , wherein the dense descriptor map comprises a high-resolution dense descriptor map and a low-resolution dense descriptor map, and the dense descriptor map calculation branch comprises a high-resolution dense descriptor map calculation sub branch and a low-resolution dense descriptor map calculation sub branch, and
wherein the determining a pair of matched pixels between the to-be-processed image and the reference image based on the dense descriptor map comprises:
determining a candidate pair of matched pixels between the to-be-processed image and the reference image based on the low-resolution dense descriptor map, wherein the low-resolution dense descriptor map is obtained by processing the to-be-processed image using the low-resolution dense descriptor map calculation sub branch; and
determining, from the candidate pair of matched pixels, the pair of matched pixels between the to-be-processed image and the reference image based on the high-resolution dense descriptor map, wherein the high-resolution dense descriptor map is obtained by processing the to-be-processed image using the high-resolution dense descriptor map calculation sub branch.
5. The method according to claim 1 , wherein the feature point classification branch comprises a plurality of feature point classification sub branches, a fusion module, and a classifier, and each of the plurality of feature point classification sub branches comprises a feature point classification extraction module, an encoder, and a fusion module, and
the inputting the to-be-processed image into the feature point classification branch to obtain a feature point category result comprises:
inputting the to-be-processed image into the plurality of feature point classification extraction modules, so as to obtain a plurality of feature point category feature maps, wherein the plurality of feature point category feature maps correspond to the plurality of feature point classification extraction modules one by one;
inputting, for each feature point category feature map of the plurality of feature point category feature maps, the feature point category feature map into the encoder so as to obtain a plurality of encoded sub feature point category feature maps, wherein the plurality of encoded sub feature point category feature maps correspond to a plurality of encoders one by one;
processing the plurality of encoded sub feature point category feature maps by using the fusion module, so as to obtain a fused feature point category map; and
processing the fused feature point category map by using the classifier, so as to obtain the feature point category result.
6. The method according to claim 1 , further comprising:
inputting the reference image into the multi-task processing model to obtain a reference feature point, a reference feature point descriptor map and a reference dense descriptor map, so as to determine the pair of matched feature points between the to-be-processed image and the reference image based on the feature point, the feature point descriptor map, the reference feature point and the reference feature point descriptor map, and determine the pair of matched pixels between the to-be-processed image and the reference image based on the dense descriptor map and the reference dense descriptor map.
7. A method of training the a multi-task processing model implemented in the method according to claim 1 , comprising:
training an initial multi-task processing model by using a training sample, so as to obtain the multi-task processing model,
wherein the training sample comprises a first sample image, a second sample image, a label for a pair of matched feature points between the first sample image and the second sample image, and a label for a pair of matched pixels between the first sample image and the second sample image, and wherein the label for the pair of matched feature points is configured to represent a matching relationship between a feature point of the first sample image and a feature point of the second sample image, and the label for the pair of matched pixels is configured to represent a matching relationship between a pixel point of the first sample image and a pixel point of the second sample image.
8. The method according to claim 7 , wherein the initial multi-task processing model comprises an initial feature point extraction branch and an initial feature point descriptor map calculation branch, and
the training an initial multi-task processing model by using a training sample so as to obtain the multi-task processing model comprises:
training the initial feature point extraction branch and the initial feature point descriptor map calculation branch by using the first sample image, the second sample image, and the label for the pair of matched feature points between the first sample image and the second sample image, so as to obtain a feature point extraction branch and a feature point descriptor map calculation branch in the multi-task processing model.
9. The method according to claim 7 , wherein the initial multi-task processing model comprises an initial dense descriptor map calculation branch, and
the training an initial multi-task processing model by using a training sample so as to obtain the multi-task processing model comprises:
training the initial dense descriptor map calculation branch by using the first sample image, the second sample image, and the label for the pair of matched pixels between the first sample image and the second sample image, so as obtain a dense descriptor map calculation branch in the multi-task processing model.
10. The method according to claim 7 , wherein the initial multi-task processing model further comprises an initial feature point classification branch, and
the training an initial multi-task processing model by using a training sample so as to obtain the multi-task processing model comprises:
training the initial feature point classification branch by using a target sample image, a feature point label of the target sample image, and a category label of the target sample image, so as to obtain a feature point classification branch of the multi-task processing model, wherein the target sample image comprises the first sample image and/or the second sample image; the feature point label of the target sample image is configured to represent a probability value that a pixel point of the target sample image is a feature point; and the category label of the target sample image is configured to represent a category of the pixel point.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, are configured to cause the at least one processor to at least:
process a to-be-processed image to obtain a feature point of the to-be-processed image, a feature point descriptor map of the to-be-processed image, and a dense descriptor map of the to-be-processed image;
determine a pair of matched feature points between the to-be-processed image and a reference image, based on the feature point and the feature point descriptor map; and
determine a pair of matched pixels between the to-be-processed image and the reference image, based on the dense descriptor map,
wherein the instructions are further configured to cause the at least one processor to at least:
input the to-be-processed image into a multi-task processing model to obtain the feature point of the to-be-processed image, the feature point descriptor map of the to-be-processed image and the dense descriptor map of the to-be-processed image,
wherein the multi-task processing model comprises a feature point classification branch, and wherein the instructions are further configured to cause the at least one processor to at least:
input the to-be-processed image into the feature point classification branch to obtain a feature point category result, so as to determine the pair of matched feature points between the to-be-processed image and the reference image based on the feature point, the feature point descriptor map and the feature point category result,
wherein the instructions are further configured to cause the at least one processor to at least:
screen the feature point based on the feature point category result, so as to determine a target feature point; and
determine the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor map, and
wherein the instructions are further configured to cause the at least one processor to at least:
extract, from the feature point descriptor map, a feature point descriptor matched with the target feature point based on the target feature point; and
determine, by using a feature point matching method, the pair of matched feature points between the to-be-processed image and the reference image based on the target feature point and the feature point descriptor matched with the target feature point.
12. The electronic device according to claim 11 , wherein the multi-task processing model further comprises a feature point extraction branch, a feature point descriptor map calculation branch and a dense descriptor map calculation branch, and wherein the instructions are further configured to cause the at least one processor to at least:
input the to-be-processed image into the feature point extraction branch to obtain the feature point;
input the to-be-processed image into the feature point descriptor map calculation branch to obtain the feature point descriptor map; and
input the to-be-processed image into the dense descriptor map calculation branch to obtain the dense descriptor map.
13. The electronic device according to claim 12 , the feature point extraction branch comprises a plurality of feature point extraction sub branches, and wherein the instructions are further configured to cause the at least one processor to at least:
input the to-be-processed image into the plurality of feature point extraction sub branches, so as to obtain a plurality of feature point scoring maps, wherein the plurality of feature point scoring maps correspond to the plurality of feature point extraction sub branches one by one;
perform a weighted summation on the plurality of feature point scoring maps to obtain a target feature point scoring map; and
perform an interpolation processing on the target feature point scoring map to obtain the feature point.
14. An electronic device, comprising:
at least one processor; and
a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, are configured to cause the at least one processor to implement the method of training the multi-task processing model according to claim 7 .