IP Library › Granted Patent US 10,783,398
Granted Patent B1
US 10,783,398 · App. 16/166,641 · Granted Sep 22, 2020

Image editor including localized editing based on generative adversarial networks

Inventors: Michael Steven Ranzinger (Boulder, CO); Nicholas Alexander Lineback (Denver, CO)
Assignee: Shutterstock, Inc.
G06K9/6215G06F16/51G06F16/54G06F16/56G06F16/5838G06F16/9535G06K9/6256G06K9/6277G06T11/60G06T2200/24
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,783,398
App. No.
16/166,641
Granted
Sep 22, 2020
Kind
B1
Abstract

A method for receiving an image query from a user via a client device is provided. The method includes determining a user personalized data based on a prior user history, generating a synthetic image with a generative tool, based on the image query and the user personalized data, and evaluating a similarity between the synthetic image and a real image in an image database with a discriminative tool. The method also includes providing the synthetic image to the user for selection and storing a user response to the synthetic image in the prior user history. A system and a non-transitory, computer readable medium storing instructions to cause the system to perform the above method are also disclosed.

Claims (40)

1. A computer-implemented method, comprising:

receiving an image query from a user via a client device;

determining a user personalized data based on a prior user history;

generating a synthetic image with a generative tool, based on the image query and the user personalized data;

evaluating a similarity between the synthetic image and a real image in an image database with a discriminative tool;

providing the synthetic image to the user for selection; and

storing a user response to the synthetic image in the prior user history,

wherein generating the synthetic image with the generative tool comprises:

selecting an image from the image database based on the image query,

identifying a centroid of a cluster in embedded space, the cluster associated with a locale information from the user personalized data, and

modifying a characteristic of the image according to a line joining a point associated with the image in the embedded space with the centroid of the cluster.

2. The computer-implemented method of claim 1 , wherein determining the user personalized data based on the prior user history comprises identifying the locale information for the user, and generating the synthetic image comprises embedding, with the generative tool, the locale information in the embedded space to classify images from the image database.

3. The computer-implemented method of claim 1 , wherein determining the user personalized data based on the prior user history comprises training a locale tool to determine the locale information for the user based on the user personalized data, and providing the locale information to the generative tool.

4. The computer-implemented method of claim 1 , wherein determining the user personalized data based on the prior user history comprises conditioning a locale tool to cluster vectors embedded with the locale information for the user in the embedded space including a locale information dimension.

5. The computer-implemented method of claim 1 , wherein determining the user personalized data based on the prior user history comprises training a binary classifier with at least one image having a likelihood of less than 50% of being selected by the user.

6. The computer-implemented method of claim 1 , wherein determining the user personalized data based on the prior user history comprises training a locale tool to determine a likelihood that an image will be downloaded by a user of a pre-selected locale.

7. The computer-implemented method of claim 1 , wherein generating the synthetic image with the generative tool comprises forming a feature vector with a locale embedding with the user personalized data, and providing the feature vector to the generative tool as a conditional context for generating the synthetic image.

8. The computer-implemented method of claim 1 , further comprising

conditioning a locale tool to cluster vectors embedded with the locale information for the user in the embedded space including a locale information dimension, wherein the locale information is determined from a client metadata.

9. The computer-implemented method of claim 1 , wherein evaluating the similarity between the synthetic image and the real image in the image database comprises selecting the real image in the image database to be a closest image to the synthetic image according to a cosine distance between the synthetic image and the real image in the image database.

10. The computer-implemented method of claim 1 , wherein storing the user response to the synthetic image in the prior user history comprises providing the user response and the synthetic image to a binary classifier of a user preference, the binary classifier comprising at least one image with a likelihood of less than 50% of being selected by the user.

11. A system comprising:

a memory, storing instructions; and

at least one processor that executes the instructions to:

receive an image query from a user via a client device;

determine a user personalized data based on a prior user history;

generate a synthetic image with a generative tool, based on the image query and the user personalized data;

evaluate a similarity between the synthetic image and a real image in an image database with a discriminative tool;

provide the synthetic image to the user for selection; and

store a user response to the synthetic image in the prior user history,

wherein to generate the synthetic image with the generative tool, the at least one processor executes the instructions to:

select an image from the image database based on the image query,

identify a centroid of a cluster in embedded space, the cluster associated with a locale information from the user personalized data, and

modify a characteristic of the image according to a line joining a point associated with the image in the embedded space with the centroid of the cluster.

12. The system of claim 11 , wherein to determine the user personalized data based on the prior user history, the one or more processors are configured to identify the locale information for the user, and generating the synthetic image comprises embedding, with the generative tool, the locale information in the embedded space to classify images from the image database.

13. The system of claim 11 , wherein to determine the user personalized data based on the prior user history, the one or more processors are configured to train a locale tool to determine the locale information for the user based on the user personalized data, and providing the locale information to the generative tool.

14. The system of claim 11 , wherein to determine the user personalized data based on the prior user history, the one or more processors are configured to condition a locale tool to cluster vectors embedded with the locale information for the user in the embedded space including a locale information dimension.

15. The system of claim 11 , wherein to determine the user personalized data based on the prior user history, the one or more processors are configured to train a binary classifier with at least one image having a likelihood of less than 50% of being selected by the user.

16. The system of claim 11 , wherein to determine the user personalized data based on the prior user history, the one or more processors are configured to train a locale tool to determine a likelihood that an image will be downloaded by a user of a pre-selected locale.

17. The system of claim 11 , wherein to generate the synthetic image with the generative tool, the one or more processors are configured to form a feature vector with a locale embedding with the user personalized data, and to provide the feature vector to the generative tool as a conditional context for generating the synthetic image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2018
From: RANZINGER, MICHAEL STEVEN; LINEBACK, NICHOLAS ALEXANDER
To: SHUTTERSTOCK, INC.
Reel/Frame 047314/0357 →
Cited By (4)
US 12,462,442 US 12,505,588 US 12,524,790 US 12,681,962