IP Library Granted Patent US 9,940,575
Granted Patent B2
US 9,940,575 · App. 14/730,476 · Granted Apr 10, 2018

Image searching

Inventor: JenHao Hsiao (Taipei, TW)
Assignee: YAHOO HOLDINGS, INC.
G06N3/08G06F17/30256
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,940,575
App. No.
14/730,476
Granted
Apr 10, 2018
Kind
B2
Abstract

As provided herein, a domain model, corresponding to a domain of an image, may be merged with a pre-trained fundamental model to generate a trained fundamental model. The trained fundamental model may comprise a feature description of the image converted into a binary code. Responsive to a user submitting a search query, a coarse image search may be performed, using a search query binary code derived from the search query, to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code. A fine image search may be performed on the candidate group utilizing a search query feature description derived from the search query. The fine image search may be used to rank images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group.

Claims (71)

1. A system for image searching, comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause implementation of an image searching component configured to:

output, from a fully connected layer of a pre-trained fundamental model, a feature description of an image;

identify a domain of the image;

merge a domain model, corresponding to the domain, into the pre-trained fundamental model to generate a trained fundamental model;

convert the feature description into a binary code using the trained fundamental model;

responsive to a user submitting a search query, perform a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images, having binary codes that exceed a threshold similarity to the search query binary code, the candidate group comprising the image having the binary code;

perform a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group, the candidate group comprising the image having the feature description; and

responsive to the image comprising a ranking above a ranking threshold, present the image to the user as a query result for the search query.

2. The system of claim 1 , the trained fundamental model comprising a convolutional neural network.

3. The system of claim 1 , the image searching component configured to:

create the trained fundamental model to comprise a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer.

4. The system of claim 1 , the image searching component configured to:

create the trained fundamental model to comprise a first fully connected layer, a second fully connected layer, and a latent layer.

5. The system of claim 4 , the image searching component configured to:

utilize the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing latent semantic content of the image.

6. The system of claim 5 , the image searching component configured to:

utilize the second fully connected layer to encode semantic information of a second query output of the latent layer.

7. The system of claim 5 , the image searching component configured to:

input the search query into the coarse image search; and

generate, using the coarse image search, the search query binary code by utilizing a second query output of the latent layer.

8. The system of claim 5 , the image searching component configured to:

input the candidate group into the fine image search; and

utilize one or more first query outputs of the first fully connected layer, using the fine image search, to rank the one or more images comprised in the candidate group.

9. The system of claim 1 , the image searching component configured to:

train the domain model to learn visual descriptors corresponding to the domain to generate learned descriptors; and

utilize the learned descriptors as feature descriptions for identifying query results.

10. The system of claim 9 , the image searching component configured to:

transfer the learned descriptors to the trained fundamental model, from the pre-trained fundamental model, utilizing back-propagation.

11. The system of claim 1 , the image searching component configured to:

train the trained fundamental model to learn binary codes from the pre-trained fundamental model and domain model.

12. A method of image searching comprising:

training a fundamental model using an image database to create a pre-trained fundamental model, the pre-trained fundamental model comprising a convolutional layer and a fully connected layer;

outputting, from the fully connected layer of the pre-trained fundamental model, a feature description of an image;

identifying a domain of the image;

merging a domain model, corresponding to the domain, with the pre-trained fundamental model to generate a trained fundamental model;

converting the feature description into a binary code using the trained fundamental model;

responsive to a user submitting a search query, performing a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code, the candidate group comprising the image having the binary code;

performing a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group, the candidate group comprising the image having the feature description; and

responsive to the image comprising a rank above a ranking threshold, presenting the image to the user as a query result for the search query.

13. The method of claim 12 , the training a fundamental model comprising:

training a convolutional neural network.

14. The method of claim 12 , comprising:

creating the trained fundamental model to comprise a first fully connected layer, a second fully connected layer, and a latent layer.

15. The method of claim 14 , comprising:

utilizing the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing a latent semantic content of the image; and

utilizing the second fully connected layer to encode semantic information of a second query output of the latent layer.

16. The method of claim 14 , the performing a course image search comprising:

inputting the search query into the coarse image search; and

generating, using the coarse image search, the search query binary code by utilizing a second query output of the latent layer.

17. The method of claim 14 , the performing a fine image search comprising:

inputting the candidate group into the fine image search; and

utilizing first query outputs of the first fully connected layer, using the fine image search, to rank the one or more images comprised in the candidate group.

18. The method of claim 12 , comprising:

training the domain model to learn visual descriptors corresponding to the domain to generate learned descriptors; and

utilizing the learned descriptors as feature descriptions for identifying query results.

19. A system for image searching, comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause implementation of an image searching component configured to:

output, from a fully connected layer of a pre-trained convolutional neural network (CNN) model, a feature description of an image;

identify a domain of the image;

merge a domain model, corresponding to the domain, with the pre-trained CNN model to generate a CNN model;

convert the feature description into a binary code using the CNN model;

responsive to a user submitting a search query, perform a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code, the candidate group comprising the image having the binary code;

perform a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group, the candidate group comprising the image having the feature description; and

responsive to the image comprising a rank above a ranking threshold, present the image to the user as a query result for the search query.

20. The system of claim 19 , the image searching component configured to:

create the trained CNN model to comprise a first fully connected layer, a second fully connected layer, and a latent layer;

utilize the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing a latent semantic content of the image; and

utilize the second fully connected layer to encode semantic information of a second query output of the latent layer.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 19, 2021
From: VERIZON MEDIA INC.
To: VERIZON PATENT AND LICENSING INC.
Reel/Frame 057453/0431 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 26, 2020
From: OATH INC.
To: VERIZON MEDIA INC.
Reel/Frame 054258/0635 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 2, 2018
From: YAHOO HOLDINGS, INC.
To: OATH INC.
Reel/Frame 045240/0310 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2017
From: YAHOO! INC.
To: YAHOO HOLDINGS, INC.
Reel/Frame 042963/0211 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 4, 2015
From: HSIAO, JENHAO
To: YAHOO!, INC.
Reel/Frame 035785/0870 →
Continuity (1)
Related Publication 20160357748A1 · Dec 8, 2016