IP Library Granted Patent US 10,198,859
Granted Patent B2
US 10,198,859 · App. 15/816,795 · Granted Feb 5, 2019

Automated three dimensional model generation

Inventors: Samuel Edward Hare (Pacific Palisades, CA); Ebony James Charlton (Santa Monica, CA); Andrew James McPhee (Culver City, CA); Michael John Evans (Venice, CA)
Assignee: Snap Inc.
G06T17/20G06K9/00201G06K9/00261G06K9/00912G06T7/55G06T2200/08G06T2200/24G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,198,859
App. No.
15/816,795
Granted
Feb 5, 2019
Kind
B2
Abstract

In various example embodiments, a system and methods are presented for generation and manipulation of three dimensional (3D) models. The system and methods cause presentation of an interface frame encompassing a field of view of an image capture device. The systems and methods detect an object of interest within the interface frame, generate a movement instruction with respect to the object of interest, and detect a first change in position and a second change in position of the object of interest. The systems and methods generate a 3D model of the object of interest based on the first change in position and the second change in position.

Claims (81)

1. A method, comprising:

causing presentation of a graphical user interface frame encompassing a graphical rendering of a field of view of an image capture device of a mobile computing device;

detecting a face within the graphical user interface frame and the field of view of the image capture device;

generating a movement instruction directing movement of the face within the field of view of the image capture device;

identifying a first key frame during a first change in position of the face within the field of view of the image capture device;

identifying one or more subsequent key frames corresponding to a second change in position of the face within the field of view of the image capture device;

based on the first key frame and the one or more subsequent key frames, generating a set of depth maps including a depth map for each key frame, the set of depth maps including a first depth map having a first resolution and a second depth map having a second resolution, the one or more subsequent key frames being identified and the set of depth maps being generated until a specified resolution is reached; and

generating a three dimensional model of the face based, at least in part, on the set of depth maps.

2. The method of claim 1 further comprising:

based on detecting the face within the graphical user interface, identifying a set of facial tracking points on the face;

in response to detecting one or more of the first change in position and the second change in position, generating graphical representations of the set of facial tracking points on the face within the graphical user interface frame; and

causing presentation of the graphical representations of the set of facial tracking points on the face for a duration of detecting the first change in position and the second change in position.

3. The method of claim 1 , wherein the first change in position of the face is an expected position change having a first initial position and a first final position, and the first change in position having a first set of intermediate positions, the first initial position, the first set of intermediate positions, and the first final position being associated with a first side of the face.

4. The method of claim 3 , wherein the second change in position of the face is an expected position change having a second initial position and a second final position, and the second change in position having a second set of intermediate positions, the second initial position, the second set of intermediate positions, and the second final position being associated with a second side of the face opposite the first side of the face.

5. The method of claim 1 , wherein detecting the first change in position further comprises:

identifying a set of facial tracking points on the face within the field of view of the image capture device;

identifying the first key frame where the set of facial tracking points have a set of first positions;

determining the first change in position of the face based on a change in position of one or more facial tracking points of the set of facial tracking points from the set of first positions; and

based on the first change in position, identifying a second key frame where the one or more facial tracking points have a second position.

6. The method of claim 5 , wherein determining the change in position of the one or more facial tracking points further comprises:

in response to initiation of the change in position of the one or more facial tracking points, identifying a trajectory for each of the one or more facial tracking points;

determining an average length of the trajectories of the one or more facial tracking points; and

determining the average length exceeds a trajectory threshold.

7. The method of claim 6 , wherein the second key frame is identified based on determining the average length exceeds the trajectory threshold.

8. The method of claim 6 further comprising:

identifying one or more subsequent key frames, including the subsequent key frame corresponding to the second change in position of the face, based on one or more changes in position of the one or more facial tracking points along the trajectories; and

based on the first key frame, the second key frame, and the one or more subsequent key frames, generating a set of relative position estimates for the mobile computing device with respect to the face, a relative position estimate of the set of relative position estimates being generated for each key frame.

9. The method of claim 8 further comprising:

based on the first key frame, the second key frame, the one or more subsequent key frames, and the set of relative position estimates, generating the set of depth maps including a depth map for each key frame.

10. The method of claim 9 , wherein the set of depth maps includes the first depth map and the second depth map, and wherein the second resolution of the second depth map is higher than the first resolution.

11. The method of claim 9 further comprising:

fusing the set of depth maps to generate the three dimensional model of the face;

defining a volumetric three dimensional grid for the three dimensional model of the face; and

representing a three dimensional surface of the three dimensional model.

12. The method of claim 11 , further comprising:

identifying a facial mesh having a set of polygons and a set of vertices connecting the set of polygons, the set of vertices representing the set of facial tracking points; and

deforming one or more portions of the mesh to fit the three dimensional model of the face by moving one or more vertices connecting two or more polygons of the set of polygons.

13. A system, comprising:

one or more processors;

an image capture device operative coupled to the one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, causes the one or more processors to perform operations comprising:

causing presentation of a graphical user interface frame encompassing a graphical rendering of a field of view of the image capture device;

detecting a face within the graphical user interface frame and the field of view of the image capture device;

generating a movement instruction directing movement of the face within the field of view of the image capture device;

identifying a first key frame during a first change in position of the face within the field of view of the image capture device;

identifying one or more subsequent key frames corresponding to a second change in position of the face within the field of view of the image capture device;

based on the first key frame and the one or more subsequent key frames, generating a set of depth maps including a depth map for each key frame, the set of depth maps including a first depth map having a first resolution and a second depth map having a second resolution, the one or more subsequent key frames being identified and the set of depth maps being generated until a specified resolution is reached; and

generating a three dimensional model of the face based, at least in part, on the set of depth maps.

14. The system of claim 13 , wherein detecting the first change in position further comprises:

identifying a set of facial tracking points on the face within the field of view of the image capture device;

identifying the first key frame where the set of facial tracking points have a set of first positions;

determining the first change in position of the face based on a change in position of one or more facial tracking points of the set of facial tracking points from the set of first positions; and

based on the first change in position, identifying a second key frame where the one or more facial tracking points have a second position.

15. The system of claim 14 , wherein determining the change in position of the one or more facial tracking points further comprises:

in response to initiation of the change in position of the one or more facial tracking points, identifying a trajectory for each of the one or more facial tracking points;

determining an average length of the trajectories of the one or more facial tracking points; and

determining the average length exceeds a trajectory threshold.

16. The system of claim 15 , wherein the operations further comprise:

identifying one or more subsequent key frames, including the subsequent key frame corresponding to the second change in position of the face, based on one or more changes in position of the one or more facial tracking points along the trajectories;

based on the first key frame, the second key frame, and the one or more subsequent key frames, generating a set of relative position estimates for the mobile computing device with respect to the face, a relative position estimate of the set of relative position estimates being generated for each key frame; and

based on the first key frame, the second key frame, the one or more subsequent key frames, and the set of relative position estimates, generating the set of depth maps including a depth map for each key frame.

17. The system of claim 16 , wherein the operations further comprise:

fusing the set of depth maps to generate the three dimensional model of the face;

defining a volumetric three dimensional grid for the three dimensional model of the face; and

representing a three dimensional surface of the three dimensional model.

18. The system of claim 17 , wherein the operations further comprise:

identifying a facial mesh having a set of polygons and a set of vertices connecting the set of polygons, the set of vertices representing the set of facial tracking points; and

deforming one or more portions of the mesh to fit the three dimensional model of the face by moving one or more vertices connecting two or more polygons of the set of polygons.

19. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by one or more processors of a mobile computing device, causes the mobile computing device to perform operations comprising:

causing presentation of a graphical user interface frame encompassing a graphical rendering of a field of view of an image capture device of a mobile computing device;

detecting a face within the graphical user interface frame and the field of view of the image capture device;

generating a movement instruction directing movement of the face within the field of view of the image capture device;

identifying a first key frame during a first change in position of the face within the field of view of the image capture device;

identifying one or more subsequent key frames corresponding to a second change in position of the face within the field of view of the image capture device;

based on the first key frame and the one or more subsequent key frames, generating a set of depth maps including a depth map for each key frame, the set of depth maps including a first depth map having a first resolution and a second depth map having a second resolution, the one or more subsequent key frames being identified and the set of depth maps being generated until a specified resolution is reached; and

generating a three dimensional model of the face based, at least in part, on the set of depth maps.

20. The non-transitory processor-readable storage medium of claim 19 , wherein detecting the first change in position further comprises:

identifying a set of facial tracking points on the face within the field of view of the image capture device;

identifying the first key frame where the set of facial tracking points have a set of first positions;

determining the first change in position of the face based on a change in position of one or more facial tracking points of the set of facial tracking points from the set of first positions; and

based on the first change in position, identifying a second key frame where the one or more facial tracking points have a second position.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 23, 2022
From: OBVIOUS ENGINEERING LIMITED
To: SNAPCHAT LIMITED
Reel/Frame 061193/0287 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 23, 2022
From: SNAP GROUP LIMITED (FORMERLY KNOWN AS SNAPCHAT LIMITED)
To: SNAP INC.
Reel/Frame 061193/0343 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 19, 2018
From: HARE, SAMUEL EDWARD; CHARLTON, EBONY JAMES; MCPHEE, ANDREW JAMES; EVANS, MICHAEL JOHN
To: OBVIOUS ENGINEERING LIMITED
Reel/Frame 047820/0204 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 19, 2018
From: OBVIOUS ENGINEERING LIMITED
To: SNAP INC.
Reel/Frame 047820/0265 →
Continuity (3)
Continuation 15080367 · Mar 24, 2016
Provisional Application 62139009 · Mar 27, 2015
Related Publication 20180075651A1 · Mar 15, 2018