Image retrieval with deep local feature descriptors and attention-based keypoint descriptors
Systems and methods of the present disclosure can use machine-learned image descriptor models for image retrieval applications and other applications. A trained image descriptor model can be used to analyze a plurality of database images to create a large-scale index of keypoint descriptors associated with the database images. An image retrieval application can provide a query image as input to the trained image descriptor model, resulting in receipt of a set of keypoint descriptors associated with the query image. Keypoint descriptors associated with the query image can be analyzed relative to the index to determine matching descriptors (e.g., by implementing a nearest neighbor search). Matching descriptors can then be geometrically verified and used to identify one or more matching images from the plurality of database images to retrieve and provide as output (e.g., by providing for display) within the image retrieval application.
1. A computing system configured to perform image retrieval, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned image descriptor model configured to determine a set of keypoint descriptors for an input image, the machine-learned image descriptor model comprising:
a feature extraction portion configured to extract a plurality of local feature descriptors from the input image; and
an attention portion configured to determine a plurality of attention scores respectively for the plurality of local feature descriptors; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a query image;
processing the query image using the feature extraction portion of the machine-learned image descriptor model to obtain, as an output of the feature extraction portion, a first plurality of local feature descriptors for the query image;
processing the first plurality of local feature descriptors for the query image using the attention portion of the machine-learned image descriptor model to obtain, as an output of the attention portion, a first plurality of attention scores respectively for the first plurality of local feature descriptors; and
determining a first set of keypoint descriptors for the query image based at least in part on the first plurality of attention scores and the first plurality of local feature descriptors, wherein the first set of keypoint descriptors correspond to a subset of the first plurality of local features.
2. The computing system of claim 1 , wherein the operations further comprise:
comparing the first set of keypoint descriptors for the query image to a plurality of reference sets of keypoint descriptors respectively associated with a plurality of database images, wherein the reference set of keypoint descriptors for each database image was generated based at least in part on respective reference local feature descriptors and reference attention scores respectively produced for such database image by the feature extraction portion and the attention portion of the machine-learned image descriptor model, and wherein the reference set of keypoint descriptors for each database image corresponds to a subset of the respective reference local feature descriptors determined for such database image.
3. The computing system of claim 2 , wherein the operations further comprise:
identifying one or more matching images based at least in part on said comparing the first set of keypoint descriptors for the query image to the plurality of reference sets of keypoint descriptors respectively associated with the plurality of database images;
retrieving the one or more matching images; and
providing the one or more matching images as a result responsive to the query image.
4. The computing system of claim 2 , wherein the reference set of keypoint descriptors for each database image comprises a predefined number of the respective local feature descriptors produced for such database image with the highest attention scores.
5. The computing system of claim 2 , wherein said comparing the first set of keypoint descriptors for the query image to the plurality of reference sets of keypoint descriptors respectively associated with the plurality of database images comprises performing a nearest neighbors search.
6. The computing system of claim 2 , wherein said comparing the first set of keypoint descriptors for the query image to the plurality of reference sets of keypoint descriptors respectively associated with the plurality of database images comprises comparing the first set of keypoint descriptors for the query image to a plurality of reduced-dimensionality reference sets of keypoint descriptors respectively associated with the plurality of database images.
7. The computing system of claim 2 , further comprising a database that stores the plurality of reference sets of keypoint descriptors respectively associated with the plurality of database images.
8. The computing system of claim 1 , wherein the feature extraction portion of the machine-learned image descriptor model comprises a convolutional neural network.
9. The computing system of claim 1 , wherein the attention portion of the machine-learned image descriptor model comprises a convolutional neural network.
10. The computing system of claim 1 , wherein determining the first set of keypoint descriptors for the query image based at least in part on the first plurality of attention scores and the first plurality of local feature descriptors comprises selecting, as the first set of keypoint descriptors, a predefined number of the plurality of local feature descriptors with the highest attention scores.
11. The computing system of claim 1 , wherein processing the query image using the feature extraction portion of the machine-learned image descriptor model comprises:
generating an image pyramid that includes respective versions of the query image at different sizes; and
respectively processing, with the feature extraction portion, the respective versions of the query image at the different sizes to obtain the first plurality of local feature descriptors.
12. The computing system of claim 1 , wherein the machine-learned image descriptor model has been trained using training images that have only image-level annotations.
13. The computing system of claim 1 , wherein the machine-learned image descriptor model has been trained through performance of a two-part training process, the two-part training process comprising:
a first training process to train the feature extraction portion of the machine-learned image descriptor model using a first loss function; and
a second training process to train the attention portion of the machine-learned image descriptor model using a second loss function.
14. A computer-implemented method to train a machine-learned image descriptor model, the method comprising:
performing, by a computing system comprising one or more computing devices, a first training process, the first training process comprising, for each of one or more first training images:
processing, by the computing system, the first training image using a feature extraction portion of the machine-learned image descriptor model to obtain, as an output of the feature extraction portion, a first plurality of local feature descriptors for the first training image;
evaluating, by the computing system, a first loss function based at least in part on the first plurality of local feature descriptors for the first training image; and
training, by the computing system, the feature extraction portion of the machine-learned image descriptor model based at least in part on the first loss function; and
performing, by the computing system, a second training process, the second training process comprising, for each of one or more second training images:
processing, by the computing system, the second training image using the feature extraction portion of the machine-learned image descriptor model to obtain, as an output of the feature extraction portion, a second plurality of local feature descriptors for the first training image;
processing the second plurality of local feature descriptors for the second training image using an attention portion of the machine-learned image descriptor model to obtain, as an output of the attention portion, a plurality of attention scores respectively for the second plurality of local feature descriptors;
evaluating, by the computing system, a second loss function based at least in part on the plurality of attention scores for the second training image; and
training, by the computing system, the attention portion of the machine-learned image descriptor model based at least in part on the second loss function.
15. The computer-implemented method of claim 14 , wherein the feature extraction portion of the machine-learned image descriptor model comprises a convolutional neural network.
16. The computer-implemented method of claim 14 , wherein training, by the computing system, the feature extraction portion of the machine-learned image descriptor model based at least in part on the first loss function comprises backpropagating, by the computing system, the first loss function through the feature extraction portion of the machine-learned image descriptor model.
17. The computer-implemented method of claim 14 , wherein the attention portion of the machine-learned image descriptor model comprises a convolutional neural network.
18. The computer-implemented method of claim 14 , wherein training, by the computing system, the attention portion of the machine-learned image descriptor model based at least in part on the second loss function comprises backpropagating, by the computing system, the second loss function through the attention portion of the machine-learned image descriptor model.
19. The computer-implemented method of claim 14 , further comprising:
after performing the first and the second training processes, using, by the computing system, the machine-learned image descriptor model to perform image retrieval.
20. One or more non-transitory computer-readable media that collectively store:
a machine-learned image descriptor model configured to determine a set of keypoint descriptors for an input image, the machine-learned image descriptor model comprising:
a feature extraction portion configured to extract a plurality of local feature descriptors from the input image;
an attention portion configured to determine a plurality of attention scores respectively for the plurality of local feature descriptors; and
a keypoint selection portion configured to select a plurality of keypoint descriptors from the plurality of local feature descriptors based at least in part on the plurality of attention scores, the plurality of keypoint descriptors corresponding to a subset of the plurality of local feature descriptors.