IP Library Granted Patent US 11,450,067
Granted Patent B2
US 11,450,067 · App. 17/301,789 · Granted Sep 20, 2022

Automated three dimensional model generation

Inventors: Samuel Edward Hare (Los Angeles, CA); Ebony James Charlton (Santa Monica, CA); Andrew James McPhee (Culver City, CA); Michael John Evans (Venice, CA)
Assignee: Snap Inc.
G06T17/20G06T7/55G06V20/64G06V40/167G06V40/67G06T2200/08G06T2200/24G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,450,067
App. No.
17/301,789
Filed
Apr 14, 2021
Granted
Sep 20, 2022
Kind
B2
Art Unit
2699
USPC
345/420
Abstract

In various example embodiments, a system and methods are presented for generation and manipulation of three dimensional (3D) models. The system and methods cause presentation of an interface frame encompassing a field of view of an image capture device. The systems and methods detect an object of interest within the interface frame, generate a movement instruction with respect to the object of interest, and detect a first change in position and a second change in position of the object of interest. The systems and methods generate a 3D model of the object of interest based on the first change in position and the second change in position.

Claims (63)

1. A method, comprising:

detecting, by one or more processors coupled to a display device, an object within a field of view of an image capture device;

generating for display a position indicator in a graphical user interface together with an image of the object that is within the field of view of the image capture device;

changing a size of the position indicator by an amount determined based on a distance of travel detected for the object, the position indicator comprising a geometric shape, the geometric shape comprising a first circular element and a second circular element that is encompassed by the first circular element, and wherein the geometric shape is reduced in size by expanding a size of the second circular element from a center of the first circular element towards an outer edge of the first circular element along a radius of the first circular element based on the distance of travel detected for the object; and

generating, by the display device, a three-dimensional model of the object based on image information of the object captured while the position indicator changes in size.

2. The method of claim 1 , further comprising:

generating for display, with the position indicator, a movement element, the movement element including one or more visual movement instructions for positioning the object within the field of view of the image capture device; and

accessing motion data representing motion of the object, wherein the size of the position indicator continuously changes based on a current position of the object within the field of view and the motion data.

3. The method of claim 1 , wherein a component moves along a straight line axis of the geometric shape from a start position towards an end position, the geometric shape being reduced in size along the straight line axis as the component moves from the start position to the end position.

4. The method of claim 3 , wherein a region between the second circular element and the first circular element is filled as the size of the second circular element is expanded.

5. The method of claim 1 , wherein a location of the position indicator is proportional to an amount of motion detected in the object, and wherein a predetermined number of key frames are identified by equally dividing a distance between an initial position and a final position of a portion of the object, further comprising:

identifying, within the predetermined number of key frames, first and second key frames corresponding respectively to first and second position changes of the object within the field of view of the image capture device relative to a starting position;

generating first and second depth maps having respectively first and second resolutions, the first depth map being generated based on the first key frame and the second depth map being generated based on the second key frame; and

generating the three-dimensional model based further on the first and second depth maps.

6. The method of claim 5 , wherein the object comprises a face, further comprising:

identifying a set of facial tracking points on the face;

in response to detecting one or more of the first and second position changes, generating graphical representations of the set of facial tracking points on the face within the graphical user interface; and

causing presentation of the graphical representations of the set of facial tracking points on the face for a duration of detecting the first and second position changes.

7. The method of claim 1 , further comprising:

identifying a set of tracking points on the object within the field of view of the image capture device;

identifying a first key frame of the key frames where the set of tracking points have a set of first positions;

determining a change in position of one or more tracking points of the set of tracking points from the set of first positions; and

based on the change in position, identifying a second key frame of the set of key frames where the one or more tracking points have a second position.

8. The method of claim 7 , wherein determining the change in position of the one or more tracking points comprises:

in response to initiation of the change in position of the one or more tracking points, identifying a trajectory for each of the one or more tracking points;

determining an average length of the trajectory of each of the one or more tracking points; and

determining the average length exceeds a trajectory threshold.

9. The method of claim 7 , further comprising:

identifying one or more subsequent key frames based on one or more changes in position of the one or more tracking points along a trajectory of each of the one or more tracking points; and

based on the first key frame, the second key frame, and the one or more subsequent key frames, generating a set of relative position estimates with respect to the object, a relative position estimate of the set of relative position estimates being generated for each key frame.

10. The method of claim 9 , further comprising:

based on the first key frame, the second key frame, the one or more subsequent key frames, and the set of relative position estimates, generating a set of depth maps including a depth map for each key frame.

11. The method of claim 10 , further comprising:

fusing the set of depth maps to generate the three-dimensional model of the object;

defining a volumetric three-dimensional grid for the three-dimensional model of the object; and

representing a three-dimensional surface of the three-dimensional model.

12. The method of claim 1 , wherein a movement element, displayed with the position indicator, overlays the image of the object, and wherein the movement element comprises a focus element and a pose element.

13. A system, comprising:

one or more processors;

an image capture device operative coupled to the one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, causes the one or more processors to perform operations comprising:

detecting an object within a field of view of an image capture device;

generating for display a position indicator in a graphical user interface together with an image of the object that is within the field of view of the image capture device;

changing a size of the position indicator by an amount determined based on a distance of travel detected for the object, the position indicator comprising a geometric shape, wherein the geometric shape comprises a first circular element and a second circular element that is encompassed by the first circular element, and wherein the geometric shape is reduced in size by expanding a size of the second circular element from a center of the first circular element towards an outer edge of the first circular element along a radius of the first circular element based on the distance of travel detected for the object; and

generating, for display, a three-dimensional model of the object based on image information of the object captured while the position indicator changes in size.

14. The system of claim 13 , the operations further comprising:

generating for display, with the position indicator, a movement element, the movement element including one or more visual movement instructions for positioning the object within the field of view of the image capture device; and

accessing motion data representing motion of the object, wherein the size of the position indicator continuously changes based on a current position of the object within the field of view and the motion data.

15. The system of claim 13 , wherein a component moves along a straight line axis of the geometric shape from a start position towards an end position, the geometric shape being reduced in size along the straight line axis as the component moves from the start position to the end position.

16. The system of claim 13 , wherein the three-dimensional model of the object is generated based on key frames identified based on the distance of travel detected for the object.

17. The system of claim 13 , wherein a location of the position indicator is proportional to an amount of motion detected in the object, and wherein a predetermined number of key frames are identified by equally dividing a distance between an initial position and a final position of a portion of the object, the operations further comprising:

identifying, within the predetermined number of key frames, first and second key frames corresponding respectively to first and second position changes of the object within the field of view of the image capture device relative to a starting position;

generating first and second depth maps having respectively first and second resolutions, the first depth map being generated based on the first key frame and the second depth map being generated based on the second key frame; and

generating the three-dimensional model based further on the first and second depth maps.

18. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by one or more processors, causes the one or more processors to perform operations comprising:

detecting an object within a field of view of an image capture device;

generating for display a position indicator in a graphical user interface together with an image of the object that is within the field of view of the image capture device;

changing a size of the position indicator by an amount determined based on a distance of travel detected for the object, the position indicator comprising a geometric shape, wherein the geometric shape comprises a first circular element and a second circular element that is encompassed by the first circular element, and wherein the geometric shape is reduced in size by expanding a size of the second circular element from a center of the first circular element towards an outer edge of the first circular element along a radius of the first circular element based on the distance of travel detected for the object; and

generating, for display, a three-dimensional model of the object based on image information of the object captured while the position indicator changes in size.

19. The non-transitory processor-readable storage medium of claim 18 , wherein a component moves along a straight line axis of the geometric shape from a start position towards an end position, the geometric shape being reduced in size along the straight line axis as the component moves from the start position to the end position.

20. The non-transitory processor-readable storage medium of claim 18 , the operations further comprising:

generating for display, with the position indicator, a movement element, the movement element including one or more visual movement instructions for positioning the object within the field of view of the image capture device; and

accessing motion data representing motion of the object, wherein the size of the position indicator continuously changes based on a current position of the object within the field of view and the motion data.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2022
From: HARE, SAMUEL EDWARD; CHARLTON, EBONY JAMES; MCPHEE, ANDREW JAMES; EVANS, MICHAEL JOHN
To: OBVIOUS ENGINEERING LIMITED
Reel/Frame 060769/0413 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2022
From: OBVIOUS ENGINEERING LIMITED
To: SNAPCHAT LIMITED
Reel/Frame 060769/0511 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2022
From: SNAP GROUP LIMITED (FORMERLY KNOWN AS SNAPCHAT LIMITED)
To: SNAP INC.
Reel/Frame 060769/0571 →
Continuity (6)
Continuation 16674892 · Nov 5, 2019
Continuation 16226108 · Dec 19, 2018
Continuation 15816795 · Nov 17, 2017
Continuation 15080367 · Mar 24, 2016
Provisional Application 62139009 · Mar 27, 2015
Related Publication 20210233310A1 · Jul 29, 2021