IP Library Granted Patent US 10,217,222
Granted Patent B2
US 10,217,222 · App. 15/332,923 · Granted Feb 26, 2019

Image cache for replacing portions of images

Inventors: Gavin James (Los Angeles, CA); Justin Lewis (Marina del Ray, CA)
Assignee: Google LLC
G06T7/11G06F17/3028G06K9/00221G06K9/00228G06T5/005G06T11/60G06K2009/00328G06T2207/10004G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,217,222
App. No.
15/332,923
Granted
Feb 26, 2019
Kind
B2
Abstract

Implementations relate to an image cache used for replacing portions of images. In some implementations, a method includes examining a received image and detecting a subject depicted in a subject portion of the received image. The method determines a likelihood that the subject portion will be used as a replacement to be inserted in place of one or more corresponding subject portions in future images examined at a later time. The subject portion, or a reference thereto, is stored in an image cache in response to the determined likelihood being greater than a predetermined threshold. The image cache is available to be searched for the subject portion and other portions of images to replace corresponding portions of future images.

Claims (58)

1. A computer-implemented method comprising:

identifying a subject face of an image, wherein the subject face depicts a subject;

comparing the subject face of the image with one or more cached faces of one or more cached images stored in an image cache;

selecting a particular cached face of the one or more cached faces to replace the subject face, wherein the selecting comprises finding the particular cached face from the one or more cached faces in the image cache based on a number of transformations to be applied to fit the particular cached face to the image being lower than a respective number of transformations to be applied to fit other faces of the one or more cached faces to the image; and

modifying the image by stitching the particular cached face in the image to replace the subject face.

2. The computer-implemented method of claim 1 ,

wherein comparing the subject face of the image with the one or more cached faces includes assigning respective scores to one or more visual image characteristics of the one or more cached faces based on comparisons of the one or more visual characteristics of the one or more cached faces with one or more corresponding visual image characteristics of the subject face, wherein the one or more visual characteristics include one or more of lighting characteristics or color characteristics, and

wherein selecting the particular cached face is further based on the respective scores.

3. The computer-implemented method of claim 1 , wherein selecting the particular cached face comprises:

based on the comparing, determining a candidate list of candidate cached faces;

assigning a score to each candidate cached face in the candidate list of candidate cached faces; and

identifying the particular cached face based on the score.

4. The computer-implemented method of claim 1 , further comprising assigning a respective score to one or more image characteristics of the subject face of the image and one or more image characteristics of the one or more cached faces stored in the image cache,

wherein the selecting the particular cached face is in response to determining that the particular cached face has at least one image characteristic associated with a better score than a score associated with a corresponding image characteristic of the subject face of the image.

5. The computer-implemented method of claim 1 further comprising:

determining whether the subject face qualifies to be stored in the image cache based on a comparison of the subject face to the one or more cached faces; and

storing the subject face in the image cache if the subject face qualifies to be stored in the image cache.

6. The computer-implemented method of claim 1 , further comprising:

determining a candidate list of one or more candidate cached faces from the image cache, wherein determining the candidate list includes determining that the one or more candidate cached faces have at least one similar characteristic to the subject face, wherein selecting the particular cached face includes selecting the particular cached face from the one or more candidate cached faces of the candidate list.

7. The computer-implemented method of claim 1 , further comprising determining that the subject face does not satisfy a quality threshold based on examination of one or more particular characteristics of the image, wherein selecting the particular cached face and stitching the particular cached face are performed in response to determining that the subject face does not satisfy the quality threshold.

8. The computer-implemented method of claim 7 wherein the one or more particular characteristics of the image include one or more facial attributes depicted in the subject face, the facial attributes including smiling mouth or eyes-open.

9. The computer-implemented method of claim 1 , wherein stitching the particular cached face comprises one or more of:

aligning the subject face and the particular cached face based on facial landmark points; or

performing gradient domain compositing.

10. The computer-implemented method of claim 1 , further comprising, prior to the stitching, transforming the particular cached face to fit the image, wherein the transforming comprises one or more of:

performing one or more of color correction or brightness correction on the particular cached face;

applying a face touch-up filter on the particular cached face; or

changing an angle of the particular cached face.

11. A system comprising:

a storage device with an image cache stored thereon; and

one or more processors configured to access the storage device and to perform operations including:

identifying a subject face of an image, wherein the subject face depicts a subject;

comparing the subject face of the image with one or more cached faces stored in the image cache;

selecting a particular cached face of the one or more cached faces, wherein the selecting is based on the comparing and is based on a number of transformations to be applied to fit the particular cached face to the image being lower than a respective number of transformations to be applied to fit other faces of the one or more cached faces to the image; and

modifying the image by stitching the particular cached face in the image to replace the subject face.

12. The system of claim 11 , wherein the operation of selecting the particular cached face comprises:

based on the comparing, determining a candidate list of candidate cached faces;

assigning a score to each candidate cached face in the candidate list of candidate cached faces; and

identifying the particular cached face based on the score.

13. The system of claim 12 , wherein the operation of determining the candidate list includes determining the candidate cached faces of the image cache to be on the candidate list, wherein the candidate cached faces include at least one of: a similar lighting to the subject face or a similar geometric orientation to the subject face.

14. The system of claim 11 , wherein the one or more processors are configured to perform further operations comprising determining that the subject face does not satisfy a quality threshold based on examination of one or more particular characteristics of the image, wherein selecting the particular cached face and stitching the particular cached face are performed in response to determining that the subject face does not satisfy the quality threshold.

15. The system of claim 14 , wherein the one or more particular characteristics of the image include one or more facial attributes depicted in the subject face, the facial attributes including smiling mouth and eyes-open.

16. The system of claim 11 , wherein the one or more processors are configured to perform further operations comprising assigning respective scores for one or more visual image characteristics of the subject face of the image and for one or more visual image characteristics of the one or more cached faces stored in the image cache, wherein the comparing includes comparing the scores for the subject face of the image with the scores for the one or more cached faces,

wherein the operation of selecting the particular cached face is further based on the scores.

17. The system of claim 11 , wherein the one or more processors are configured to perform further operations comprising:

determining whether the subject face qualifies to be stored in the image cache based on a comparison of the subject face to the one or more cached faces; and

storing the subject face in the image cache if the subject face qualifies to be stored in the image cache.

18. The system of claim 11 , wherein stitching the particular cached face comprises one or more of:

aligning the subject face and the particular cached face based on facial landmark points; or

performing gradient domain compositing.

19. A non-transitory computer readable medium having stored thereon program instructions that, when executed by a processor, cause the processor to perform operations comprising:

identifying a subject face of an image, wherein the subject face depicts a subject;

comparing the subject face of the image with one or more cached faces stored in an image cache;

selecting a particular cached face of the one or more cached faces, wherein the selecting is based on the comparing and is based on a number of transformations to be applied to fit the particular cached face to the image being lower than a respective number of transformations to be applied to fit other faces of the one or more cached faces to the image; and

modifying the image by stitching the particular cached face in the image to replace the subject face.

20. The non-transitory computer readable medium of claim 19 ,

wherein the operation of comparing the subject face of the image with the one or more cached faces includes assigning respective scores to one or more visual image characteristics of the one or more cached faces based on comparisons of the one or more visual characteristics of the one or more cached faces with one or more corresponding visual image characteristics of the subject face, wherein the one or more visual characteristics include one or more of lighting characteristics or color characteristics, and

wherein the operation of selecting the particular cached face is further based on the respective scores.

Assignments (2)
CHANGE OF NAME Recorded Dec 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044695/0115 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2017
From: JAMES, GAVIN; LEWIS, JUSTIN
To: GOOGLE INC.
Reel/Frame 043746/0420 →
Continuity (3)
Continuation 14523517 · Oct 24, 2014
Provisional Application 61896648 · Oct 28, 2013
Related Publication 20170103538A1 · Apr 13, 2017