Granted Patent B2

US 12,450,868 · App. 17/683,777 · Granted Oct 21, 2025

Inventors: Shuai Mou (Guangdong, CN); Wan Peng Xiao (Guangdong, CN); Qi Ju (Guangdong, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G06V10/757G06F16/532G06F16/55G06V10/462G06V10/761G06V10/762G06V10/764G06V10/774G06V10/776G06V10/82

Abstract

Claims (92)

1. An image retrieval method, performed by at least one processor of an image retrieval device, and comprising:

obtaining an image;

performing feature extraction on the image according to a first feature extraction manner to obtain a first feature, the first feature extraction manner being a deep learning-based extraction manner and the first feature being a depth feature;

determining, based on the first feature of the image, at least one candidate image matching the image;

performing feature extraction on the image and each of the at least one candidate image according to a second feature extraction manner to obtain a second feature, the second feature extraction manner being a non-deep learning-based extraction manner;

sequentially performing key point matching processing on the image with each of the at least one candidate image based on the extracted second feature;

determining a first image, from among the at least one candidate image, as an image retrieval result, based on a quantity of matched key points between the first image and the image being greater than a quantity threshold; and

transmitting the image retrieval result.

2. The image retrieval method according to claim 1 , wherein the performing feature extraction on the image according to the first feature extraction manner comprises performing feature extraction on the image based on a deep learning model, the deep learning model being obtained through training based on a first training dataset and a second training dataset, and

wherein the first training dataset is a dataset provided with a label, and the second training dataset is a dataset not provided with a label.

3. The image retrieval method according to claim 2 , wherein the deep learning model comprises a convolutional neural network (CNN), a clustering branch, and a classification branch, and

wherein a training process of the deep learning model comprises:

inputting sample images comprised in the first training dataset and the second training dataset into the CNN;

invoking a clustering model of the clustering branch;

iteratively clustering feature data outputted by the CNN;

outputting an obtained clustering result as a pseudo-label to the classification branch;

invoking a classifier of the classification branch;

obtaining a loss value based on the pseudo-label; and

iteratively updating a weight of the CNN according to the loss value.

4. The image retrieval method according to claim 3 , wherein the obtaining the loss value based on the pseudo-label comprises obtaining the loss value based on the pseudo-label according to the pseudo-label and a classification result of the classifier for the feature data outputted by the CNN.

5. The image retrieval method according to claim 3 , further comprising:

obtaining normalized mutual information (NMI) of clustering results of two adjacent rounds, the NMI indicating stability of the deep learning model, and a larger value of the NMI indicating that the deep learning model is closer to a convergence state;

determining the convergence state of the deep learning model according to the NMI; and

stopping the training process after the deep learning model reaches the convergence state, a convergence condition of the deep learning model being that the value of the NMI reaches a stable value.

6. The image retrieval method according to claim 5 , wherein the determining, based on the first feature of the image, the at least one candidate image comprises:

obtaining feature distances between the first feature of the image and first features stored in a feature database;

sorting retrieval images corresponding to the stored first features in descending order of the feature distances; and

determining a specified quantity of retrieval images ranked top as the at least one candidate image.

7. The image retrieval method according to claim 5 , wherein the performing feature extraction on each of the at least one candidate image comprises:

obtaining a first feature of each of the at least one candidate image;

obtaining a feature distance between the first feature of each of the at least one candidate image and the first feature of the image; and

performing second feature extraction on a second image in the at least one candidate image, and

wherein a feature distance between the second image and the image is less than a first distance threshold.

8. The image retrieval method according to claim 5 , wherein the obtaining the image comprises:

receiving an image retrieval request transmitted by a terminal; and

obtaining the image from the image retrieval request, and

wherein the image retrieval method further comprises, after obtaining the image retrieval result, transmitting the image retrieval result to the terminal.

9. The image retrieval method according to claim 5 , wherein the performing feature extraction on the image and each of the at least one candidate image according to the second feature extraction manner comprises:

performing, for any image on which feature extraction is to be performed, key point detection on the image, to obtain a plurality of key points; and

establishing a descriptor for each detected key point separately, the descriptor being represented by using a feature vector of a fixed dimension.

10. The image retrieval method according to claim 9 , wherein the sequentially performing key point matching processing on the image with each of the at least one candidate image based on the extracted second feature comprises:

obtaining a first key point descriptor set of the image and a second key point descriptor set of any candidate image;

determining a feature distance between any a first key point descriptor and a second key point descriptor, the first key point descriptor being from the first key point descriptor set, and the second key point descriptor being from the second key point descriptor set;

determining matched key points in the image and the candidate image based on the obtained feature distance, a feature distance corresponding to any pair of matched key points being less than a second distance threshold; and

filtering the matched key points to obtain final matched key points.

11. The image retrieval method according to claim 5 , wherein the second feature is a scale-invariant feature transform (SIFT) feature.

12. An image retrieval device, comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

first obtaining code configured to cause at least one of the at least one processor to obtain an image;

first performing code configured to cause at least one of the at least one processor to perform feature extraction on the image according to a first feature extraction manner to obtain a first feature, the first feature extraction manner being a deep learning-based extraction manner and the first feature being a depth feature;

first determining code configured to cause at least one of the at least one processor to determine, based on the first feature of the image, at least one candidate image matching the image;

second performing code configured to cause at least one of the at least one processor to perform feature extraction on the image and each of the at least one candidate image according to a second feature extraction manner to obtain a second feature, the second feature extraction manner being a non-deep learning-based extraction manner;

third performing code configured to cause at least one of the at least one processor to sequentially perform key point matching processing on the image with each of the at least one candidate image based on the extracted second feature;

second determining code configured to cause at least one of the at least one processor to determine a first image, from among the at least one candidate image, as an image retrieval result based on a quantity of matched key points between the first image and the image being greater than a quantity threshold; and

transmitting code configured to cause at least one of the at least one processor to transmit the image retrieval result.

13. The image retrieval device according to claim 12 , wherein the first performing code is further configured to cause at least one of the at least one processor to perform feature extraction on the image based on a deep learning model, the deep learning model being obtained through training based on a first training dataset and a second training dataset, and

wherein the first training dataset is a dataset provided with a label, and the second training dataset is a dataset not provided with a label.

14. The image retrieval device according to claim 13 , wherein the deep learning model comprises a convolutional neural network (CNN), a clustering branch, and a classification branch, and

wherein the program code further comprises training code configured to cause at least one of the at least one processor to perform a training process of the deep learning model, the training process comprising:

inputting sample images comprised in the first training dataset and the second training dataset into the CNN;

invoking a clustering model of the clustering branch;

iteratively clustering feature data outputted by the CNN;

outputting an obtained clustering result as a pseudo-label to the classification branch;

invoking a classifier of the classification branch;

obtaining a loss value based on the pseudo-label; and

iteratively updating a weight of the CNN according to the loss value.

15. The image retrieval device according to claim 14 , wherein the obtaining the loss value based on the pseudo-label comprises:

obtaining the loss value based on the pseudo-label according to the pseudo-label and a classification result of the classifier for the feature data outputted by the CNN.

16. The image retrieval device according to claim 14 , wherein the program code further comprises:

second obtaining code configured to cause at least one of the at least one processor to obtain normalized mutual information (NMI) of clustering results of two adjacent rounds, the NMI indicating stability of the deep learning model, and a larger value of the NMI indicating that the deep learning model is closer to a convergence state;

determining code configured to cause at least one of the at least one processor to determine the convergence state of the deep learning model according to the NMI; and

stopping code configured to cause at least one of the at least one processor to stop the training process after the deep learning model reaches the convergence state, a convergence condition of the deep learning model being that the value of the NMI reaches a stable value.

17. The image retrieval device according to claim 16 , wherein the first determining code is further configured to cause at least one of the at least one processor to:

obtain feature distances between the first feature of the image and first features stored in a feature database;

sort retrieval images corresponding to the stored first features in descending order of the feature distances; and

determine a specified quantity of retrieval images ranked top as the at least one candidate image.

18. The image retrieval device according to claim 16 , wherein the first performing code is further configured to cause at least one of the at least one processor to:

obtain a first feature of each candidate image;

obtain a feature distance between the first feature of each of the at least one candidate image and the first feature of the image; and

perform second feature extraction on a second image in the at least one candidate image, and

wherein a feature distance between the second image and the image is less than a first distance threshold.

19. The image retrieval device according to claim 16 , wherein the obtaining code is further configured to cause at least one of the at least one processor to receive an image retrieval request transmitted by a terminal and obtain the image from the image retrieval request, and

wherein the transmitting code is further configured to cause at least one of the at least one processor to transmit the image retrieval result to the terminal.

20. A non-transitory computer readable storage medium, storing at least one instruction executable by at least one processor to perform an image retrieval method comprising:

obtaining an image;

performing feature extraction on the image according to a first feature extraction manner to obtain a first feature, the first feature extraction manner being a deep learning-based extraction manner and the first feature being a depth feature;

determining, based on the first feature of the image, at least one candidate image matching the image;

performing feature extraction on the image and each of the at least one candidate image according to a second feature extraction manner to obtain a second feature, the second feature extraction manner being a non-deep learning-based extraction manner;

sequentially performing key point matching processing on the image with each of the at least one candidate image based on the extracted second feature;

determining a first image, from among the at least one candidate image, as an image retrieval result, based on a quantity of matched key points between the first image and the image being greater than a quantity threshold; and

transmitting the image retrieval result.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (27)