IP Library Granted Patent US 9,721,188
Granted Patent B2
US 9,721,188 · App. 14/683,643 · Granted Aug 1, 2017

Landmarks from digital photo collections

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,721,188
App. No.
14/683,643
Granted
Aug 1, 2017
Kind
B2
Abstract

Methods and systems for automatic detection of landmarks in digital images and annotation of those images are disclosed. A method for detecting and annotating landmarks in digital images includes the steps of automatically assigning a tag descriptive of a landmark to one or more images in a plurality of text-associated digital images to generate a set of landmark-tagged images, learning an appearance model for the landmark from the set of landmark-tagged images, and detecting the landmark in a new digital image using the appearance model. The method can also include a step of annotating the new image with the tag descriptive of the landmark.

Claims (32)

1. A method to detect landmarks in digital images, comprising:

assigning, by one or more processors, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark feature, wherein the text tag is based upon one or more n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored based on image-name links between the one or more n-grams and the one or more images, wherein each of the image-name links is a variable indicative of whether the one or more n-grams are included in text tags of the one or more images;

building an appearance model for the landmark feature based on the assigned text tag descriptive of the landmark feature; and

detecting the landmark feature in a different image using the appearance model.

2. The method of claim 1 , further comprising annotating the different image with the text tag descriptive of the landmark feature.

3. The method of claim 1 , wherein assigning the text tag descriptive of the landmark feature further comprises generating a list of the one or more n-grams.

4. The method of claim 3 , wherein assigning the text tag further comprises computing an n-gram score for one or more of the one or more n-grams.

5. The method of claim 4 , wherein the text tag descriptive of the landmark feature is based on one or more of the one or more n-grams.

6. The method of claim 4 , wherein assigning the text tag further comprises assigning correlation weights to each of the plurality of text-associated digital images based on correlation of text associated with each image with text associated with one or more other images in the plurality of text-associated digital images.

7. The method of claim 6 , further comprising generating a matching-images graph from the plurality of text associated digital images and assigning a match score to each edge in the matching-images graph, wherein the match score represents a level of match between two images connected by the each edge.

8. The method of claim 7 , wherein the n-grams are scored based on the matching-images graph.

9. The method of claim 7 , further comprising generating the image-name links between the one or more n-grams and the plurality of text-associated digital images based upon the correlation weights and the matching images graph.

10. The method of claim 7 , further comprising estimating a geo-reliability score for one or more images in the plurality of text-associated digital images based on the matching-images graph.

11. A system to detect landmarks in digital images, comprising:

one or more processors; and

a non-transitory computer-readable medium having instructions stored thereon, the instructions executable by the one or more processors to:

assign to one or more images in a plurality of text-associated digital images a text tag descriptive of a landmark object, wherein the text tag is based upon one or more n-grams from one or more texts associated with the one or more images that are scored based on image-name links between the one or more n-grams and the one or more images, wherein the image-name links are variables indicative of whether the one or more n-grams are included in text tags of the one or more images;

learn an appearance model for the landmark object from the assigned text tag;

detect the landmark object in a new image using the appearance model; and

annotate the new image with the assigned text tag descriptive of the landmark object.

12. The system of claim 11 , wherein the instructions further comprise instructions to generate a list of the one or more n-grams from the one or more texts associated with the one or more images.

13. The system of claim 12 , wherein the instructions further comprise instructions to compute an n-gram score for one or more of the one or more n-grams from the one or more texts.

14. The system of claim 13 , wherein the text tag descriptive of the landmark object is assigned to the one or more images based on the n-gram score meeting a threshold.

15. The system of claim 13 , wherein the instructions further comprise instructions to assign correlation weights to the plurality of text-associated digital images based on correlation of text associated with each image with other images in the plurality of text-associated digital images.

16. The system of claim 15 , wherein the instructions further comprise instructions to generate a matching-images graph from the plurality of text associated digital images and assign a match score to each edge in the images graph representing a level of match between two images connected by each edge.

17. The system of claim 16 , wherein the n-grams are scored based at least in part on the matching-images graph.

18. The system of claim 16 , wherein the instructions further comprise instructions to generate the image-name links between the one or more n-grams and the plurality of text-associated digital images based upon the correlation weights and the matching images graph.

19. The system of claim 16 , wherein the instructions further comprise instructions to estimate a geo-reliability score for one or more images in the plurality of text-associated digital images based on the matching-images graph.

20. A method to detect landmarks in digital images, comprising:

assigning, by one or more processors, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark feature, wherein the text tag is based upon one or more n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored based on image-name links between the one or more n-grams and the one or more images and wherein the assigning comprises assigning correlation weights to each of the plurality of text-associated digital images based on correlation of text associated with each image with text associated with one or more other images in the plurality of text-associated digital images;

building an appearance model for the landmark feature based on the assigned text tag descriptive of the landmark feature; and

detecting the landmark feature in a different image using the appearance model.

Assignments (2)
CHANGE OF NAME Recorded Dec 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044695/0115 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2017
From: ADAM, HARTWIG; ZHANG, LI
To: GOOGLE INC.
Reel/Frame 043746/0276 →