IP Library › Granted Patent US 10,991,110
Granted Patent B2
US 10,991,110 · App. 16/844,095 · Granted Apr 27, 2021

Methods and systems to modify a two dimensional facial image to increase dimensional depth and generate a facial image that appears three dimensional

Inventors: Paul D. Yan (Novato, CA); Thitikun Vorachart (San Rafael, CA)
Assignee: Activision Publishing, Inc.
G06T7/50G06K9/00208G06K9/00268G06K9/621G06T11/00G06T15/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,991,110
App. No.
16/844,095
Granted
Apr 27, 2021
Kind
B2
Abstract

The specification describes methods and systems for increasing a dimensional depth of a two-dimensional image of a face to yield a face image that appears three dimensional. The methods and systems identify key points on the 2-D image, obtain a texture map for the 2-D image, determines one or more proportions within the 2-D image, and adjusts the texture map of the 3-D model based on the determined one or more proportions within the 2-D image.

Claims (40)

1. A computer-implemented method for increasing a dimensional depth of a two-dimensional image to yield an augmented reality (AR) face mask, said method being implemented in a computer having a processor and a random access memory, wherein said processor is in data communication with a display and with a storage unit, the method comprising:

acquiring from the storage unit the two-dimensional image;

acquiring an image of a face of a person from a camera;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, identifying a first plurality of key points on the two-dimensional image;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, identifying a second plurality of key points on the two-dimensional image;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, generating a texture map of the two-dimensional image;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, projecting said texture map of the two-dimensional image onto the image of the face of the person;

using said computer and executing a plurality of programmatic instructions stored in the storage unit, modifying the first plurality of key points based on the second plurality of key points;

and

using said computer, outputting the AR face mask image based on the modified first plurality of key points.

2. The computer-implemented method of claim 1 , wherein each of the first plurality or second plurality of key points include points representative of a plurality of anatomical locations on the face, wherein said anatomical locations include points located on the eyebrows, eyes, nose, and lips.

3. The computer-implemented method of claim 1 , wherein the texture map comprises a plurality of non-overlapping, triangular regions.

4. The computer-implemented method of claim 1 , further comprising, using said computer and executing a plurality of programmatic instructions stored in the storage unit, determining a first set of one or more proportions within the two-dimensional image, determining a second set of one or more proportions within the image of the face of the person, and determining a plurality of scaling factors, wherein each of said scaling factors is a function of one of said first set of one or more proportions and a corresponding one of said second set of one or more proportions and wherein each of said plurality of scaling factors is a ratio of one of said first set of one or more proportions to the corresponding one of said second set of one or more proportions.

5. The computer-implemented method of claim 1 , further comprising, using said computer and executing a plurality of programmatic instructions stored in the storage unit, determining a first set of one or more proportions within the two-dimensional image and determining a second set of one or more proportions within the image of the face of the person, wherein the determining the first set of one or more proportions within the two-dimensional image comprises determining proportions from measurements between at least two anatomical positions on the face of the person.

6. The computer-implemented method of claim 1 , further comprising, using said computer and executing a plurality of programmatic instructions stored in the storage unit, determining a first set of one or more proportions within the two-dimensional image and determining a second set of one or more proportions within the image of the face of the person, wherein the determining a first set of one or more proportions within the two-dimensional image comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

7. The computer-implemented method of claim 6 , wherein the first anatomical distance is at least one of a lateral face width, a lateral jaw width, a lateral temple width, a lateral eyebrow width, a lateral chin width, a lateral lip width, and a lateral nose width and wherein the second anatomical distance is a distance between two temples of the face.

8. The computer-implemented method of claim 6 , wherein the first anatomical distance is at least one of a vertically defined lip thickness, a vertical distance between a nose and a nose bridge, a vertical distance between a lip and a nose bridge, a vertical distance between a chin and a nose bridge, a vertical eye length, and a vertical distance between a jaw and a nose bridge and wherein the second anatomical distance is at least one of a distance between two anatomical positions on said face and a distance between two temples of the face.

9. The computer-implemented method of claim 6 , wherein the first anatomical distance is a distance between two anatomical positions on said face and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the face and a point located proximate a right edge of a right eyebrow of the face.

10. The computer-implemented method of claim 1 , wherein the determining a second set of one or more proportions within the image of the face of the person comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

11. The computer-implemented method of claim 10 , wherein the first anatomical distance is at least one of a lip thickness, a distance between a nose and a nose bridge, a distance between a lip and a nose bridge, a distance between a chin and a nose bridge, an eye length, and a distance between a jaw and a nose bridge of the image of the face of the person and wherein the second anatomical distance is a distance between two anatomical positions on said image of the face of the person.

12. The computer-implemented method of claim 10 , wherein the first anatomical distance is a distance between two anatomical positions on said image of the face of the person and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the image of the face of the person and a point located proximate a right edge of a right eyebrow of the image of the face of the person.

13. A computer readable non-transitory medium comprising a plurality of executable programmatic instructions wherein, when said plurality of executable programmatic instructions are executed by a processor, a process for increasing a dimensional depth of a two-dimensional image to yield an augmented reality (AR) face mask is performed, said plurality of executable programmatic instructions comprising:

programmatic instructions, stored in said computer readable non-transitory medium, for acquiring from the storage unit the two-dimensional image;

programmatic instructions, stored in said computer readable non-transitory medium, for identifying a first plurality of key points on the two-dimensional image;

programmatic instructions, stored in said computer readable non-transitory medium, for identifying a second plurality of key points on the two-dimensional image;

programmatic instructions, stored in said computer readable non-transitory medium, for generating a texture map of the two-dimensional image;

programmatic instructions, stored in said computer readable non-transitory medium, for translating said texture map of the two-dimensional image onto an image of a face of a person acquired from a camera;

programmatic instructions, stored in said computer readable non-transitory medium, for modifying the first plurality of key points based on the second plurality of key points;

programmatic instructions, stored in said computer readable non-transitory medium, for outputting the AR face mask image based on the modified first plurality of key points.

14. The computer readable non-transitory medium of claim 13 , wherein each of the first plurality or second plurality of key points include points representative of a plurality of anatomical locations on the face, wherein said anatomical locations include points located on the eyebrows, eyes, nose, and lips.

15. The computer readable non-transitory medium of claim 13 , wherein the texture map comprises a plurality of non-overlapping, triangular regions.

16. The computer readable non-transitory medium of claim 13 , further comprising programmatic instructions, stored in said computer readable non-transitory medium, for determining a first set of one or more proportions within the two-dimensional image and determining a second set of one or more proportions within the image of the face of the person, wherein the determining the first set of one or more proportions within the two-dimensional image comprises determining proportions from measurements between at least two anatomical positions on the face of the person.

17. The computer readable non-transitory medium of claim 13 , further comprising programmatic instructions, stored in said computer readable non-transitory medium, for determining a first set of one or more proportions within the two-dimensional image, for determining a second set of one or more proportions within the image of the face of the person, and for determining a plurality of scaling factors, wherein each of said scaling factors is a function of one of said first set of one or more proportions and a corresponding one of said second set of one or more proportions and wherein each of said plurality of scaling factors is a ratio of one of said first set of one or more proportions to the corresponding one of said second set of one or more proportions.

18. The computer readable non-transitory medium of claim 13 , further comprising programmatic instructions, stored in said computer readable non-transitory medium, for determining a first set of one or more proportions within the two-dimensional image and determining a second set of one or more proportions within the image of the face of the person, wherein the determining a first set of one or more proportions within the two-dimensional image comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

19. The computer readable non-transitory medium of claim 18 , wherein the first anatomical distance is at least one of a lateral face width, a lateral jaw width, a lateral temple width, a lateral eyebrow width, a lateral chin width, a lateral lip width, and a lateral nose width and wherein the second anatomical distance is a distance between two temples of the face.

20. The computer readable non-transitory medium of claim 18 , wherein the first anatomical distance is at least one of a vertically defined lip thickness, a vertical distance between a nose and a nose bridge, a vertical distance between a lip and a nose bridge, a vertical distance between a chin and a nose bridge, a vertical eye length, and a vertical distance between a jaw and a nose bridge and wherein the second anatomical distance is a distance between two temples of the face.

21. The computer readable non-transitory medium of claim 18 , wherein the first anatomical distance is a distance between two anatomical positions on said face and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the face and a point located proximate a right edge of a right eyebrow of the face.

22. The computer readable non-transitory medium of claim 13 , wherein the determining a second set of one or more proportions within the image of the face of the person comprises determining a first anatomical distance and dividing said first anatomical distance by a second anatomical distance.

23. The computer readable non-transitory medium of claim 22 , wherein the first anatomical distance is at least one of a lip thickness, a distance between a nose and a nose bridge, a distance between a lip and a nose bridge, a distance between a chin and a nose bridge, an eye length and a distance between a jaw and a nose bridge of the image of the face of the person and wherein the second anatomical distance is a distance between two anatomical positions on said three-dimensional mesh image of the face of the person.

24. The computer readable non-transitory medium of claim 22 , wherein the first anatomical distance is a distance between two anatomical positions on said image of the face of the person and the second anatomical distance is a distance between a point located proximate a left edge of a left eyebrow of the image of the face of the person and a point located proximate a right edge of a right eyebrow of the image of the face of the person.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 9, 2020
From: YAN, PAUL D.; VORACHART, THITIKUN
To: ACTIVISION PUBLISHING, INC.
Reel/Frame 052353/0939 →
Continuity (3)
Continuation 16041529 · Jul 20, 2018
Continuation 15370166 · Dec 6, 2016
Related Publication 20200302623A1 · Sep 24, 2020