Automatic face annotation of images contained in media content
Face-containing images within web pages are automatically annotated to identify the people having those faces. The annotation is based on faces detected in the images and named entities detected in text associated with the images. Each candidate named entity may be scored by the prominence of the named entity in the text relative to the other extracted named entities. Queries are sent to a search engine based on the extracted candidate named entities. Sample images are returned. Face similarity calculations may be computed based on the featured faces and sample faces detected in the search engine-returned sample images to associate a probability score between each featured face and each candidate named entity. A bipartite matching instance may be solved to arrive at a maximum likelihood assignment of named entities to featured faces.
1. A computer-implemented method, comprising:
detecting one or more faces in one or more first digital images;
identifying one or more named entities contained in text of a document that is associated with the one or more first digital images;
for each particular named entity of the one or more named entities:
sending, to a search engine, a query based on the particular named entity; and
receiving, from the search engine, in response to execution of the query by the search engine, one or more second digital images;
calculating a plurality of scores, wherein the plurality of scores includes a single score for each pairing of a set of pairings, wherein the set of pairings includes a pairing between each face of the one or more faces detected in the one or more first digital images and each named entity of the one or more named entities;
wherein each score of the plurality of scores indicates a similarity between a particular face of the one or more faces and the one or more second digital images received from the search engine in response to execution of a query based on a particular named entity of the one or more named entities;
storing, in a non-transitory storage medium, a mapping, based at least in part on the plurality of scores, between (1) each face of a subset of the faces detected in the one or more first digital images and (2) a particular named entity of the one or more named entities; and
wherein the method is performed by one or more special-purpose computing devices.
2. The method of claim 1 , wherein the detecting one or more faces in one or more first digital images further comprises filtering the faces based on an image region size threshold associated with image regions detected as containing a face in the one or more first digital images.
3. The method of claim 1 , further comprising:
for each particular named entity of the one or more named entities, calculating a relevance score associated with the particular named entity based on one or more of: a frequency of occurrence of the particular named entity in the text, a relative order in which the particular named entity appears in the text, and formatting information associated with the named entity in the text.
4. The method of claim 3 , wherein each score of the plurality of scores is based at least in part on a relevance score.
5. The method of claim 1 , wherein the text of the document that is associated with the one or more first digital images includes one or more of: text displayed on a web page containing the one or more first digital images, a Uniform Resource Locator (URL), and markup language associated with the web page.
6. The method of claim 1 , wherein the identifying one or more named entities further comprises filtering the named entities based on determining whether one or more of the named entities are included in a dictionary or a taxonomy.
7. The method of claim 1 , wherein receiving the one or more second digital images from the search engine further comprises filtering the one or more second digital images based on one or more of: image size, image quality, and image data size.
8. The method of claim 1 , further comprising:
detecting one or more sample faces in the one or more second digital images;
filtering the one or more second digital images based on one or more of[M]: images containing two or more sample faces, detected sample face image region size, image size, image quality, and image data size.
9. The method of claim 8 , wherein each score of the plurality of scores is generated based on a multi-label classifier that takes as input the faces detected in the first digital images and sample faces detected in the second digital images.
10. The method of claim 1 , further comprisingmodifying the mapping based on one or more post-filtering processes.
11. The method of claim 1 , further comprising:
for a particular face of the one or more faces detected in the one or more first digital images, calculating a second score for an arbitrary named entity that is not contained in the text of the document;
based on the second score, modifying a particular mapping between the particular face and a particular named entity of the one or more named entities.
12. The method of claim 1 , wherein the mapping is computed based on the plurality of scores and a maximum likelihood assignment between the one or more faces detected in the one or more first digital images and the one or more named entities.
13. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform:
detecting one or more faces in one or more first digital images;
identifying one or more named entities contained in text of a document that is associated with the one or more first digital images;
for each particular named entity of the one or more named entities:
sending, to a search engine, a query based on the particular named entity; and
receiving, from the search engine, in response to execution of the query by the search engine, one or more second digital images;
calculating a plurality of scores, wherein the plurality of scores includes a single score for each pairing of a set of pairings, wherein the set of pairings includes a pairing between each face of the one or more faces detected in the one or more first digital images and each named entity of the one or more named entities;
wherein each score of the plurality of scores indicates a similarity between a particular face of the one or more faces and the one or more second digital images received from the search engine in response to execution of a query based on a particular named entity of the one or more named entities;
storing, in a non-transitory storage medium, a mapping, based at least in part on the plurality of scores, between (1) each face of a subset of the faces detected in the one or more first digital images and (2) a particular named entity of the one or more named entities.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the detecting one or more faces in one or more first digital images further comprises filtering the faces based on an image region size threshold associated with image regions detected as containing a face in the one or more first digital images.
15. The non-transitory computer-readable storage medium of claim 13 , further comprising:
for each particular named entity of the one or more identified named entities, calculating a relevance score associated with the particular named entity based on one or more of: a frequency of occurrence of the particular named entity in the text, a relative order in which the particular named entity appears in the text, and formatting information associated with the named entity in the text.
16. The non-transitory computer-readable storage medium of claim 15 , wherein each score of the plurality of scores is based at least in part on a relevance score.
17. The non-transitory computer-readable storage medium of claim 13 , wherein the text of the document that is associated with the one or more first digital images includes one or more of: text displayed on a web page containing the one or more first digital images, a Uniform Resource Locator (URL), and markup language associated with the web page.
18. The non-transitory computer-readable storage medium of claim 13 , wherein the identifying one or more named entities further comprises filtering the named entities based on determining whether one or more of the named entities are included in a dictionary or a taxonomy.
19. The non-transitory computer-readable storage medium of claim 13 , wherein receiving the one or more second digital images from the search engine further comprises filtering the one or more second digital images based on one or more of: image size, image quality, and image data size.
20. The non-transitory computer-readable storage medium of claim 13 , further comprising:
detecting one or more sample faces in the one or more second digital images;
filtering the one or more second digital images based on one or more of: images containing two or more sample faces, detected sample face image region size, image size, image quality, and image data size.
21. The non-transitory computer-readable storage medium of claim 20 , wherein each score of the plurality of scores is generated based on a multi-label classifier that takes as input the faces detected in the first digital images and sample faces detected in the second digital images.
22. The non-transitory computer-readable storage medium of claim 13 , further comprising modifying the mapping based on one or more post-filtering processes.
23. The non-transitory computer-readable storage medium of claim 22 , further comprising:
for a particular face of the one or more faces detected in the one or more first digital images, calculating a second score for an arbitrary named entity that is not contained in the text of the document;
based on the second score, modifying a particular mapping between the particular face and a particular named entity of the one or more named entities.
24. The non-transitory computer-readable storage medium of claim 13 , wherein the mapping is computed based on the plurality of scores and a maximum likelihood assignment between the one or more faces detected in the one or more first digital images and the one or more named entities.