IP Library Granted Patent US 9,495,614
Granted Patent B1
US 9,495,614 · App. 14/633,704 · Granted Nov 15, 2016

Verifying labels for images using image recognition

Inventors: Robert C. Boman (Thousand Oaks, CA); John Flynn (Venice, CA); Farhana Bandukwala (San Clemente, CA)
Assignee: Google Inc.
G06K9/6215G06F17/30241G06K9/6267G06K9/68
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,495,614
App. No.
14/633,704
Filed
Feb 27, 2015
Granted
Nov 15, 2016
Kind
B1
Art Unit
2668
USPC
382/218
Abstract

Implementations relate to verifying labels for images using image recognition. In some implementations, a method includes obtaining an image associated with location information, obtaining one or more descriptor labels associated with the location information, determining one or more recognized image features depicted in the image, comparing the recognized image features with the descriptor labels, and determining one or more verified labels from the one or more descriptor labels. The verified labels are determined to describe at least one of the one or more recognized image features depicted in the image based on the comparing of the recognized image features with the descriptor labels.

Claims (53)

1. A method comprising:

obtaining, using at least one processor, an image associated with geographic location information, the geographic location information indicative of an associated geographic location;

obtaining, using at least one processor, one or more descriptor labels associated with the geographic location information;

determining, using at least one processor, one or more recognized image features depicted in the image, wherein the one or more recognized image features depict one or more geographic features of the geographic location, wherein the recognized image features are recognized in the image based on one or more object recognition techniques, wherein the one or more recognized image features are associated with one or more confidence scores, each of the one or more confidence scores indicating an amount of confidence in the recognition of the associated image features;

comparing, using at least one processor, the one or more recognized image features with the one or more descriptor labels; and

determining, using at least one processor, one or more verified labels from the one or more descriptor labels, wherein determining one or more verified labels is based at least in part on the one or more confidence scores, wherein the one or more verified labels are verified to describe at least one of the one or more recognized image features depicted in the image based on the comparing of the one or more recognized image features with the one or more descriptor labels.

2. The method of claim 1 further comprising storing, using at least one processor, the one or more verified labels as associated with the image.

3. The method of claim 1 further comprising causing, using at least one processor, at least one of the descriptor labels that do not describe at least one of the one or more recognized image features to be unassociated with the image.

4. The method of claim 1 wherein the location information includes Global Positioning System (GPS) location information associated with the image as metadata.

5. The method of claim 1 wherein the one or more descriptor labels originated from one or more data sources that store the one or more descriptor labels in association with the location information.

6. The method of claim 1 wherein the descriptor labels include at least one of: a proper name of a geographic feature, and a generic descriptive term for a geographic feature, and wherein the one or more geographic features include at least one of: a monument, a landmark, a building, and a landscape feature.

7. The method of claim 1 wherein comparing the one or more recognized image features with the one or more descriptor labels includes comparing one or more recognized labels associated with the one or more recognized image features with the one or more descriptor labels, and

wherein determining one or more verified labels from the one or more descriptor labels includes checking for one or more matches between the one or more descriptor labels and the one or more recognized labels, wherein each matching descriptor label is a verified label.

8. The method of claim 1 further comprising determining an importance score for at least one of the one or more recognized image features based on at least one of:

a size of the at least one recognized image feature relative to an area of the image; and

a type of the at least one recognized image feature,

wherein the determining of the one or more verified labels is based at least in part on the importance score.

9. The method of claim 1 wherein determining one or more verified labels from the one or more descriptor labels includes checking for one or more matches between the one or more descriptor labels and the one or more recognized image features, wherein each matching descriptor label is a verified label, and

wherein the matches are determined based on checking for a predetermined threshold amount of similarity found between the one or more descriptor labels and the one or more recognized image features, wherein the predetermined threshold amount of similarity is checked based on at least one similarity score indicating an amount of similarity between the one or more descriptor labels and the one or more recognized image features.

10. The method of claim 1 further comprising:

examining, using at least one processor, the one or more descriptor labels without examining content of the image to determine whether one or more possible discrepancies are present between at least one of the descriptor labels and the content of the image, wherein the determining whether one or more possible discrepancies are present includes at least one of:

determining whether the one or more descriptor labels are a plurality of descriptor labels,

determining whether at least one of the one or more descriptor labels have an associated confidence score that does not satisfy a threshold; and

determining whether at least one of the one or more descriptor labels have a particular type or are present on a stored list of labels,

wherein the determining of the one or more recognized image features is performed only in response to determining that the one or more possible discrepancies are present.

11. The method of claim 1 wherein determining one or more verified labels includes determining a plurality of verification levels for the one or more descriptor labels, wherein at least one of the plurality of verification levels indicates a less confident verification than at least one other of the plurality of verification levels.

12. The method of claim 7 further comprising checking, using at least one processor, at least one of the descriptor labels that do not match at least one of the one or more recognized image features for one or more predetermined characteristics or conditions allowing the at least one descriptor label to be associated with the image, wherein the one or more predetermined characteristics or conditions include at least one of:

the at least one descriptor label describes a higher-level geographic feature that includes at least one of the one or more recognized image features; and

the at least one descriptor label is associated with at least one of the matching descriptor labels.

13. The method of claim 1 further comprising causing, using at least one processor, the image and the one or more verified labels to be available as a search result for searches input by a user for information in the one or more descriptor labels.

14. A system comprising:

a storage device; and

at least one processor accessing the storage device and operative to perform operations comprising:

obtaining an image associated with geographic location information;

obtaining one or more descriptor labels associated with the geographic location information;

determining one or more recognized image features depicted in the image, wherein the one or more recognized image features are associated with one or more confidence scores, each of the one or more confidence scores indicating an amount of confidence in the recognition of the associated image features;

comparing the one or more recognized image features with the one or more descriptor labels;

determining one or more verified labels from the one or more descriptor labels, wherein determining one or more verified labels is based at least in part on the one or more confidence scores, wherein the one or more verified labels are determined to describe at least one of the one or more recognized image features depicted in the image based on the comparing of the one or more recognized image features with the one or more descriptor labels; and

causing at least one of the one or more descriptor labels that do not describe at least one of the one or more recognized image features to be unassociated with the image.

15. The system of claim 14 wherein comparing the one or more recognized image features with the one or more descriptor labels includes comparing one or more recognized labels associated with the one or more recognized image features with the one or more descriptor labels, and further comprising:

storing the one or more verified labels as associated with the image.

16. The system of claim 14 wherein the one or more descriptor labels originated from one or more data sources that store the one or more descriptor labels in association with the location information, and further comprising:

sending an unreliability indication to the one or more data sources, wherein the unreliability indication is associated with the at least one descriptor label that does not describe the at least one recognized image feature.

17. A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to perform operations including:

obtaining an image associated with geographic location information;

obtaining one or more descriptor labels associated with the geographic location information;

determining one or more recognized image features depicted in the image, wherein the one or more recognized image features are associated with one or more confidence scores, each of the one or more confidence scores indicating an amount of confidence in the recognition of the associated image features;

comparing the one or more recognized image features with the one or more descriptor labels;

determining verified labels from the one or more descriptor labels, wherein determining one or more verified labels is based at least in part on the one or more confidence scores, wherein the verified labels are determined to describe at least one of the one or more recognized image features depicted in the image based on the comparing of the one or more recognized image features with the one or more descriptor labels;

causing the one or more verified labels to be associated with the image; and

causing at least one of the descriptor labels that do not describe at least one of the one or more recognized image features to be unassociated with the image.

18. The computer readable medium of claim 17 wherein comparing the one or more recognized image features with the one or more descriptor labels includes comparing one or more recognized labels associated with the one or more recognized image features with the one or more descriptor labels, and

wherein the geographic location information is indicative of an associated geographic location, and wherein the one or more recognized image features depict one or more geographic features of the geographic location.

Assignments (2)
CHANGE OF NAME Recorded Oct 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044129/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 27, 2015
From: BOMAN, ROBERT; FLYNN, JOHN; BANDUKWALA, FARHANA
To: GOOGLE INC.
Reel/Frame 035052/0034 →