Using relevance feedback in face recognition

1. A computer-implemented method for recognizing faces within images, the method comprising:

accessing at least one query face;

determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face;

providing an indication of the search result images to a user;

receiving feedback related to the accuracy of the search result images;

inferring additional feedback based on the received feedback; and

updating the set of search result images based on the received feedback and the inferred additional feedback,

wherein receiving feedback related to the accuracy of the search result images comprises receiving implicit user feedback provided when the user interacts with the provided search result images in at least one of a natural manner and a manner that implies that the application-selected face is, or is not, a match for the query face through a failure to interact with one or more of the provided search result images, and

wherein a natural manner comprises a user printing one or more of the provided search result images, and inferring additional feedback based on the received feedback comprises inferring that a face in the printed one or more of the provided search result images matches the query face.

2. The method of claim 1 wherein determining a set of search result images includes:

creating a query feature vector for the query face;

forming a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces;

determining distances between the query feature vector and the search feature vectors;

selecting one or more search feature vectors that are within a selected distance of the query feature vector; and

designating images that include faces that correspond to the selected search feature vectors as the set of search result images.

3. The method of claim 2 wherein forming a set of search feature vectors includes:

detecting a class for the query face;

determining faces in a set of faces that have the same class as the query face; and

forming the set of search feature vectors by selecting feature vectors that correspond to the determined faces.

4. The method of claim 2 wherein determining distances between the query feature vector and the search feature vectors comprises determining a weighted Euclidean distance between the query feature vector and each of the search feature vectors.

5. The method of claim 2 wherein creating the query feature vector comprises:

locating the query face within a query image;

detecting facial features of the query face;

warping the query face based on the detected facial features;

extracting information regarding the detected facial features from the warped face; and

forming the query feature vector based on the extracted information.

6. The method of claim 5 further comprising:

detecting non-face based features from the query image;

extracting information regarding the non-face based features from the query image; and

forming the query feature vector based on the extracted information regarding the detected non-face based features and the extracted information regarding the detected facial features.

7. The method of claim 2 wherein:

accessing at least one query face comprises accessing multiple query faces;

creating a query feature vector for the query face comprises creating a query feature vector for each of the multiple faces; and

determining distances between the query feature vector and the search feature vectors includes determining distances between the multiple query feature vectors and the search feature vectors.

8. The method of claim 2 further comprising:

identifying negative faces that are determined to not be the same as the query face; and

creating negative feature vectors for the negative faces, wherein:

accessing at least one query face comprises accessing multiple query faces;

creating a query feature vector for the query face comprises creating a query feature vector for each of the multiple faces; and

determining distances between the query feature vector and the search feature vectors includes determining distances between the multiple query feature vectors and the search feature vectors and determining distances between the negative feature vectors and the search feature vectors.

9. The method of claim 1 wherein the inferred additional feedback includes information about faces in the search result images other than the application-selected face.

10. The method of claim 1 wherein the inferred additional feedback includes information about faces in the search result images other than a face selected by the user as being the same as the query face.

11. The method of claim 1 further comprising:

accessing a set of images that include faces that have not been labeled;

sorting the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and

accessing the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.

12. The method of claim 1 wherein the search result images comprise images within an online photograph album.

13. The method of claim 12 wherein the indication of the search result images provided to the user is an indication that the search result images have been associated with the query face, wherein the query face is in a labeled image.

14. The method of claim 12 wherein the indication of the search result images provided to the user includes the search result images.

15. The method of claim 1 wherein the search result images comprise images within a social network.

16. A non-transitory computer-useable medium storing a computer program, the computer program including instructions for causing a computer to perform the following operations:

access at least one query image that includes at least one query face;

determine a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face;

provide an indication of the search result images to a user;

receive feedback related to the accuracy of the search result images;

infer additional feedback based on the received feedback; and

update the set of search result images based on the received feedback and the inferred additional feedback,

wherein the instructions for causing a computer to receive feedback related to the accuracy of the search result images comprise instructions for causing a computer to receive implicit user feedback provided when the user interacts with the provided search result images in at least one of a natural manner and a manner that implies that the application-selected face is, or is not, a match for the query face through a failure to interact with one or more of the provided search result images, and

wherein a natural manner comprises a user printing one or more of the provided search result images, and the instructions for causing a computer to infer additional feedback based on the received feedback comprise instructions for causing a computer to infer that a face in the printed one or more of the provided search result images matches the query face.

17. The computer-useable medium of claim 16 wherein the instructions for causing the computer to determine a set of search result images include instructions for causing the computer to:

create a query feature vector for the query face;

form a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces;

determine distances between the query feature vector and the search feature vectors;

select one or more search feature vectors that are within a selected distance of the query face feature vector; and

designate images that include faces that correspond to the selected search feature vectors as the set of search result images.

18. The computer-useable medium of claim 17 wherein the instructions for causing the computer to determine distances between the query feature vector and the search feature vectors include instructions for causing the computer to determine a weighted Euclidean distance between the query face feature vector and each of the search feature vectors.

19. The computer useable medium of claim 16 wherein the inferred additional feedback includes information about faces in the search result images other than the application-selected face or information about faces in the search result images other than a face selected by the user as being the same as the query face.

20. The computer useable medium of claim 16 further includes instructions for causing the computer to:

access a set of images that include faces that have not been labeled;

sort the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and

access the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.

21. The computer-useable medium of claim 17 wherein:

the instructions for causing the computer to access at least one query face include instructions for causing the computer to access multiple query faces;

the instructions for causing the computer to create a query feature vector for the query face include instructions for causing the computer to create a query feature vector for each of the multiple faces; and

the instructions for causing the computer to determine distances between the query feature vector and the search feature vectors include instructions for causing the computer to determine distances between the multiple query feature vectors and the search feature vectors.

22. The computer-useable medium of claim 17 further includes instructions for causing the computer to:

identify negative faces that are determined to not be the same as the query face; and

create negative feature vectors for the negative faces,

wherein:

the instructions for causing the computer to access at least one query face includes instructions for causing the computer to access multiple query faces;

the instructions for causing the computer to create a query feature vector for the query face include instructions for causing the computer to create a query feature vector for each of the multiple faces; and

the instructions for causing the computer to determine distances between the query feature vector and the search feature vectors include instructions for causing the computer to determine distances between the multiple query feature vectors and the search feature vectors and determine distances between the negative feature vectors and the search feature vectors.

23. A system comprising: one or more processing devices; one or more storage devices storing instructions that, when implemented by the one or more processing devices, causes the one or more processing devices to implement:

a feature extractor configured to:

access at least one query face,

create a query feature vector for the query face, and

form a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces;

a face classifier configured to:

determine distances between the query feature vector and the search feature vectors,

select one or more search feature vectors that are within a selected distance of the query feature vector, and

designate images that include faces that correspond to the selected search feature vectors as a set of search result images;

an interface manager configured to:

provide an indication of the search result images to a user, and

receive feedback related to the accuracy of the search result images; and

a feedback manager configured to infer additional feedback based on the received feedback,

wherein the face classifier is also configured to update the set of search result images based on the received feedback and the inferred additional feedback,

wherein receiving feedback related to the accuracy of the search result images comprises receiving implicit user feedback provided when the user interacts with the provided search result images in at least one of a natural manner and a manner that implies that the application-selected face is, or is not, a match for the query face through a failure to interact with one or more of the provided search result images, and

wherein a natural manner comprises a user printing one or more of the provided search result images, and inferring additional feedback based on the received feedback comprises inferring that a face in the printed one or more of the provided search result images matches the query face.

24. The method of claim 1 , wherein inferring additional feedback comprises inferring extra information for the other faces within one or more of the provided search result images for which the user has not provided explicit or implicit feedback.