IP Library Granted Patent US 10,726,086
Granted Patent B2
US 10,726,086 · App. 15/351,978 · Granted Jul 28, 2020

Aesthetic search engine

Inventors: Chen Longbin (Palo Alto, CA); Le Kang (Palo Alto, CA); Lei Yao (Palo Alto, CA); Zhenyu Mao (Palo Alto, CA)
Assignee: Houzz, Inc.
G06F16/9535G06F16/248G06F16/24578G06F16/285G06F16/40G06F16/50G06F16/51G06F16/583G06F16/5854G06F16/5866G06K9/00624G06K9/00711G06K9/00718G06K9/00744G06K9/4628
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,726,086
App. No.
15/351,978
Granted
Jul 28, 2020
Kind
B2
Abstract

An improved approach for returning aesthetically relevant search results is disclosed. A training set of images (e.g., designer-created images) is used to train a detection engine that detects items in the images as features. A classification engine is configured to analyze the features and generate classification indices for the features. A user can select an item, and the classification index for the feature corresponding to the item is retrieved. The classification index is used to identify result images, which can be returned ranked according user action data and other parameters, such as style.

Claims (68)

1. A method comprising:

receiving a set of images from a first client device of a designer user of a network site, each image displaying items grouped in the image by the designer user, the items being physical items and the image being a photograph of the physical items as grouped by the designer user;

generating, using a detection neural network, an initial feature dataset for the items by applying an initial detection model of the detection neural network to the set of images;

generating, using a classification neural network, classifications for initial feature data items in the initial feature dataset by applying a classification model of the classification neural network to the initial feature dataset;

merging a portion of the set of images into a visually distinguishable dataset based on images in the portion having similar classifications generated by the classification neural network;

generating an updated detection model by training the detection neural network on the images in the visually distinguishable dataset;

generating, using the detection neural network, feature data for the items by applying the updated detection model to the set of images, the feature data specifying shape attributes of the items displayed in the set of images;

generating using the classification neural network, classification indices for the items by applying the classification model to the feature data;

causing, on a second client device of a non-designer user of the network site, a presentation displaying one of the set of images received from the designer user;

receiving, from the second client device of the non-designer user, a selection of an item in the one of the set of images;

determining a subset of the set of images that display items having classification indices closest to a classification index of the item;

generating a search result that ranks the subset of images according to user action data that describes user bookmarks, by users of the network site, of images in the subset; and

causing a presentation of the search result on a display device of the second client device of the non-designer user.

2. The method of claim 1 , wherein the detection neural network and the classification neural network are separate convolutional neural networks.

3. The method of claim 2 , further comprising:

receiving, from other client devices of the users of the network site, the user bookmarks of one or more of the images; and

storing the user bookmarks of the one or more images.

4. The method of claim 1 , wherein the subset of images are ranked in greatest-to-least order.

5. The method of claim 1 , further comprising:

receiving, from the second client device, selection of a grouping parameter, the grouping parameter specifying an attribute of the images; and

wherein each of the images in the search result has an attribute value that is the same as the attribute specified by the grouping parameter.

6. The method of claim 5 , wherein the attribute includes one or more of the following: style, designer, color, geographical location.

7. The method of claim 5 , wherein the grouping parameter is specified by the non-designer user through a user interface.

8. The method of claim 1 , wherein the item is selected from one of the images of the set of images through the display device of the second client device.

9. A system comprising:

one or more processors of a machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:

receiving a set of images from a first client device of a designer user of a network site, each image displaying items grouped in the image by the designer user, the items being physical items and the image being a photograph of the physical items as grouped by the designer user;

generating, using a detection neural network, an initial feature dataset for the items by applying an initial detection model of the detection neural network to the set of images;

generating, using a classification neural network, classifications for initial feature data items in the initial feature dataset by applying a classification model of the classification neural network to the initial feature dataset;

merging a portion of the set of images into a visually distinguishable dataset based on images in the portion having similar classifications generated by the classification neural network;

generating an updated detection model by training the detection neural network on the images in the visually distinguishable dataset;

generating, using the detection neural network, feature data for the items by applying the updated detection model to the set of images, the feature data specifying shape attributes of the items displayed in the set of images;

generating, using the classification neural network, classification indices for the items by applying the classification model to the feature data;

causing, on a second client device of a non-designer user of the network site, a presentation displaying one of the set of images received from the designer user;

receiving, from the second client device of the non-designer user, a selection of an item in the one of the set of images;

determining a subset of the set of images that display items having classification indices closest to a classification index of the item;

generating a search result that ranks the subset of images according to user action data that describes user bookmarks, by users of the network site, of images in the subset; and

causing a presentation of the search result on a display device of the second client device of the non-designer user.

10. The system of claim 9 , wherein the detection neural network and the classification neural network are separate convolutional neural networks.

11. The system of claim 10 , the operations further comprising:

receiving, from other client devices of the users of the network site, the user bookmarks of one or more of the images; and

storing the user bookmarks of the one or more images.

12. The system of claim 9 , wherein the subset of images are ranked in greatest-to-least order.

13. The system of claim 9 , the operations further comprising:

receiving, from the second client device, selection of a grouping parameter, the grouping parameter specifying an attribute of the images; and

wherein each of the images in the search result has an attribute value that is the same as the attribute specified by the grouping parameter.

14. The system of claim 13 , wherein the attribute includes one or more of the following: style, designer, color, geographical location.

15. The system of claim 13 , wherein the grouping parameter is specified by the non-designer user through a user interface.

16. The system of claim 9 , wherein the item is selected from one of the images of the set of images through the display device of the second client device.

17. A non-transitory machine-readable storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

receiving a set of images from a first client device of a designer user of a network site, each image displaying items grouped in the image by the designer user, the items being physical items and the image being a photograph of the physical items as grouped by the designer user;

generating, using a detection neural network, an initial feature dataset for the items by applying an initial detection model of the detection neural network to the set of images;

generating, using a classification neural network, classifications for initial feature data items in the initial feature dataset by applying a classification model of the classification neural network to the initial feature dataset;

merging a portion of the set of images into a visually distinguishable dataset based on images in the portion having similar classifications generated by the classification neural network;

generating an updated detection model by training the detection neural network on the images in the visually distinguishable dataset;

generating, using the detection neural network, feature data for the items by applying the updated detection model to the set of images, the feature data specifying shape attributes of the items displayed in the set of images;

generating, using the classification neural network, classification indices for the items b T applying the classification model to the feature data;

causing, on a second client device of a non-designer user of the network site, a presentation displaying one of the set of images received from the designer user;

receiving, from the second client device of the non-designer user, a selection of an item in the one of the set of images;

determining a subset of the set of images that display items having classification indices closest to a classification index of the item;

generating a search result that ranks the subset of images according to user action data that describes user bookmarks, by users of the network site, of images in the subset; and

causing a presentation of the search result on a display device of the second Bent device of the non-designer user.

18. The non-transitory machine-readable storage medium of claim 17 , wherein the detection neural network and the classification neural network are separate convolutional neural networks.

19. The non-transitory machine-readable storage medium of claim 18 , the operations further comprising:

receiving, from other client devices of the users of the network site, the user bookmarks of one or more of the images; and

storing the user bookmarks of the one or more images.

20. The non-transitory machine-readable storage medium of claim 17 , wherein the subset of images are ranked in greatest-to-least order.

Assignments (3)
RELEASE OF SECURITY INTEREST Recorded Feb 18, 2022
From: HERCULES CAPITAL, INC., AS COLLATERAL AND ADMINISTRATIVE AGENT
To: HOUZZ INC.
Reel/Frame 059191/0501 →
INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Nov 5, 2019
From: HOUZZ INC.
To: HERCULES CAPITAL, INC., AS COLLATERAL AND ADMINISTRATIVE AGENT
Reel/Frame 050928/0333 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 20, 2017
From: CHEN, LONGBIN; KANG, LE; YAO, LEI; MAO, ZHENYU
To: HOUZZ, INC.
Reel/Frame 043639/0412 →
Continuity (1)
Related Publication 20180137201A1 · May 17, 2018