IP Library Granted Patent US 10,719,763
Granted Patent B2
US 10,719,763 · App. 16/392,305 · Granted Jul 21, 2020

Image searching

Inventor: JenHao Hsiao (Taipei, TW)
Assignee: Oath Inc.
G06N3/08G06F16/5838G06N3/0427G06N3/0454G06N3/084Y04S10/54
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,719,763
App. No.
16/392,305
Granted
Jul 21, 2020
Kind
B2
Abstract

As provided herein, a domain model, corresponding to a domain of an image, may be merged with a pre-trained fundamental model to generate a trained fundamental model. The trained fundamental model may comprise a feature description of the image converted into a binary code. Responsive to a user submitting a search query, a coarse image search may be performed, using a search query binary code derived from the search query, to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code. A fine image search may be performed on the candidate group utilizing a search query feature description derived from the search query. The fine image search may be used to rank images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group.

Claims (53)

1. A system for image searching, comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause implementation of an image searching component configured to:

convert a feature description of an image into a binary code using a trained fundamental model;

responsive to a user submitting a search query, perform a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images; and

perform a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group, the candidate group comprising the image having the feature description.

2. The system of claim 1 , the image searching component configured to:

responsive to the image comprising a ranking above a ranking threshold, present the image to the user as a query result for the search query.

3. The system of claim 1 , the image searching component configured to:

create the trained fundamental model to comprise a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer.

4. The system of claim 1 , the image searching component configured to:

create the trained fundamental model to comprise a first fully connected layer, a second fully connected layer, and a latent layer.

5. The system of claim 4 , the image searching component configured to:

utilize the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing latent semantic content of the image.

6. The system of claim 5 , the image searching component configured to:

utilize the second fully connected layer to encode semantic information of a second query output of the latent layer.

7. The system of claim 5 , the image searching component configured to:

input the search query into the coarse image search; and

generate, using the coarse image search, the search query binary code by utilizing a second query output of the latent layer.

8. The system of claim 5 , the image searching component configured to:

input the candidate group into the fine image search; and

utilize one or more first query outputs of the first fully connected layer, using the fine image search, to rank the one or more images comprised in the candidate group.

9. The system of claim 1 , the trained fundamental model comprising a convolutional neural network.

10. The system of claim 1 , the image searching component configured to:

transfer learned descriptors to the trained fundamental model, from a pre-trained fundamental model, utilizing back-propagation.

11. The system of claim 1 , the image searching component configured to:

train the trained fundamental model to learn binary codes from a pre-trained fundamental model and a domain model.

12. A method of image searching comprising:

converting a feature description of an image into a binary code;

responsive to a user submitting a search query, performing a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images; and

performing a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group, the candidate group comprising the image having the feature description.

13. The method of claim 12 , comprising:

responsive to the image comprising a rank above a ranking threshold, presenting the image to the user as a query result for the search query.

14. The method of claim 12 , comprising:

creating a trained fundamental model to comprise at least one of a first fully connected layer, a second fully connected layer, or a latent layer, the converting performed using the trained fundamental model.

15. The method of claim 14 , comprising:

utilizing the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing a latent semantic content of the image; and

utilizing the second fully connected layer to encode semantic information of a second query output of the latent layer.

16. The method of claim 14 , the performing a course image search comprising:

inputting the search query into the coarse image search; and

generating, using the coarse image search, the search query binary code by utilizing a second query output of the latent layer.

17. The method of claim 14 , the performing a fine image search comprising:

inputting the candidate group into the fine image search; and

utilizing first query outputs of the first fully connected layer, using the fine image search, to rank the one or more images comprised in the candidate group.

18. The method of claim 12 , the converting performed using a convolutional neural network.

19. A system for image searching, comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause implementation of an image searching component configured to:

convert a feature description of an image into a binary code using a trained fundamental model;

identify a candidate group, comprising one or more images, having binary codes corresponding to a second binary code; and

perform a fine image search on the candidate group utilizing a second feature description to rank the one or more images within the candidate group, the candidate group comprising the image having the feature description.

20. The system of claim 19 , the image searching component configured to:

responsive to the image comprising a rank above a ranking threshold, present the image to a user.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 4, 2023
From: YAHOO HOLDINGS, INC.
To: OATH INC.
Reel/Frame 065748/0549 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 1, 2023
From: HSIAO, JENHAO
To: YAHOO! INC.
Reel/Frame 065737/0036 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 1, 2023
From: YAHOO! INC.
To: YAHOO HOLDINGS, INC.
Reel/Frame 065737/0578 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 19, 2021
From: VERIZON MEDIA INC.
To: VERIZON PATENT AND LICENSING INC.
Reel/Frame 057453/0431 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 26, 2020
From: OATH INC.
To: VERIZON MEDIA INC.
Reel/Frame 054258/0635 →
Continuity (3)
Continuation 15948061 · Apr 9, 2018
Continuation 14730476 · Jun 4, 2015
Related Publication 20190251438A1 · Aug 15, 2019