Content-aware ranking for visual search

1. A computer-implemented method comprising:

accessing training data including:

at least one query;

a set of images associated with the at least one query;

and

for an individual image of the set of images, one or more user-assigned labels indicating a degree of relevance of the individual image to the at least one query, the degree of relevance selected from a plurality of degrees of relevance;

ranking the set of images associated with the at least one query based at least in part on the respective one or more user-assigned labels thereby providing a first relevance ranking of the set of images;

extracting visual features from the set of images;

extracting textual features associated with the set of images;

ranking the set of images using a ranking model that jointly uses the visual features and the textual features thereby providing a second relevance ranking of the set of images;

comparing the first relevance ranking of the set of images to the second relevance ranking of the set of images; and

updating the ranking model based in part on ranking differences produced by the comparing.

2. The computer-implemented method of claim 1 , wherein the ranking model includes a parameter utilized to balance relevance of the textual features against relevance of the visual features.

3. The computer-implemented method of claim 1 , wherein the ranking model employs a sparse graph utilizing a k-nearest neighbor strategy to determine a similarity between a pair of images based at least in part on the visual features.

4. The computer-implemented method of claim 1 , wherein updating the ranking model comprises utilizing a cutting plane algorithm that determines a constraint set.

5. The computer-implemented method of claim 4 , wherein the constraint set is determined a least in part by finding a most violated prediction for the at least one query.

6. One or more device memories storing computer-executable instructions that, when executed via a processor, perform the method of claim 1 .

7. A computer-implemented method comprising:

ranking a set of images associated with at least one training query based at least in part on user-assigned labels indicating a degree of relevance of an individual image to the at least one training query, the ranking providing a first relevance ranking of the set of images;

ranking, based at least in part on visual features and textual features associated with the set of images, the set of images using a ranking model thereby providing a second relevance ranking of the set of images;

comparing the first relevance ranking of the set of images to the second relevance ranking of the set of images;

updating the ranking model based in part on ranking differences produced by the comparing;

receiving an image search query;

retrieving a plurality of candidate images relevant to the image search query; and

ranking, based at least in cart on the ranking model, at least a portion of the plurality of candidate images.

8. The computer-implemented method of claim 7 , wherein the ranking model employs a sparse graph utilizing a k-nearest neighbor strategy to determine a similarity between a pair of candidate images based on visual features extracted from the pair of candidate images.

9. The computer-implemented method of claim 7 , wherein:

the ranking model is generated utilizing a cutting plane algorithm that determines a constraint set; and

the constraint set is determined by finding a most violated prediction for the at least one training query.

10. One or more device memories storing computer-executable instructions that, when executed via a processor, perform the method of claim 7 .

11. The computer-implemented method of claim 7 , further comprising:

extracting visual features from the at least the portion of the plurality of candidate images; and

determining textual features associated with the at least the portion of the plurality of candidate images, wherein the ranking model jointly incorporates a use of the textual features and a use of the visual features to rank the at least the portion of the plurality of candidate images, wherein the ranking model comprises a parameter utilized to balance relevance of the textual features against relevance of the visual features.

12. A computing device comprising:

at least one processor;

a memory coupled to the processor storing computer-executable instructions that, when executed by the at least one processor, configure the computing device to:

rank a set of images associated with at least one training query based at least in part on user-assigned labels indicating a degree of relevance of an individual image to the at least one training query, the ranking providing a first relevance ranking of the set of images;

rank, based at least in part on visual features and textual features associated with the set of images, the set of images using a ranking model thereby providing a second relevance ranking of the set of images;

compare the first relevance ranking of the set of images to the second relevance ranking of the set of images; and

update the ranking model based in part on ranking differences produced by the comparing.

13. The computing device of claim 12 , wherein the ranking model optimizes one or more parameters by minimizing ranking loss based at least in part on the comparing, the one or more parameters being optimized based on a constraint set associated with a most violated prediction for the at least one training query.

14. The computing device of claim 12 , wherein the ranking model employs a sparse graph utilizing a k-nearest neighbor strategy to determine a similarity between a pair of images based on the visual features.

15. The computing device of claim 12 , wherein the ranking model comprises a parameter utilized to balance relevance of the textual features against relevance of the visual features.