Sculpting augmented reality content using gestures in a messaging system
The subject technology detects from a set of frames, a first gesture, the first gesture corresponding to a pinch gesture. The subject technology detects a first location and a first position of a first representation of a first finger from the first gesture and a second location and a second position of a second representation of a second finger from the first gesture. The subject technology detects a first collision event corresponding to a first collider and a second collider intersecting with a third collider of a first virtual object. The subject technology detects a first change in the first location and the first position and a second change in the second location and the second position. The subject technology modifies the first virtual object to include an additional augmented reality content based at least in part on the first change and the second change.
1 . A method, comprising:
detecting, using one or more hardware processors, from a set of frames, a first gesture, the first gesture corresponding to a pinch gesture;
detecting, using the one or more hardware processors, a first location and a first position of a first representation of a first finger from the first gesture and a second location and a second position of a second representation of a second finger from the first gesture, the first location comprising a first set of spatial coordinates of the first representation of the first finger in three dimensional (3D) space, the second location comprising a second set of spatial coordinates of the second representation of the second finger in the 3D space, the first position comprising a first orientation or first pose of the first representation of the first finger, and the second position comprising a second orientation or second pose of the second representation of the second finger;
detecting, using the one or more hardware processors, a first collision event corresponding to a first collider of the first representation of the first finger and a second collider of the second representation of the second finger intersecting with a third collider of a first virtual object to initiate selection of a first portion of the first virtual object as a first starting point for generating first additional augmented reality content that extends from the first starting point;
detecting, using the one or more hardware processors, a first change in the first location and the first position and a second change in the second location and the second position;
in response to the first change and the second change, modifying, using the one or more hardware processors, the first virtual object to include an additional augmented reality content based at least in part on the first change and the second change;
rendering, using the one or more hardware processors, the first virtual object based at least in part on the modifying within a first scene;
providing for display, using the one or more hardware processors, the rendered first virtual object within the first scene,
wherein the rendered first virtual object is provided for display within a messaging application executing on a first client device; and
transmitting, using the one or more hardware processors and via the messaging application, the rendered first virtual object to a second client device of a recipient user of the messaging application.
2 . The method of claim 1 , wherein the first change and the second change does not result in an increase in a distance greater than a threshold value between the first representation of the first finger and the second representation of the second finger, and the first finger and the second finger is from a same hand.
3 . The method of claim 1 , further comprising:
detecting from a second set of frames a third location and a third position of the first representation of the first finger and a fourth location and a fourth position of the second representation of the second finger;
detecting a first particular change in the third location and the third position and a second particular change in the fourth location and the fourth position, wherein the first particular change and the second particular change results in an increase in a distance greater than a threshold value between the first representation of the first finger and the second representation of the second finger, and the first finger and the second finger is from a same hand; and
in response to detecting the first particular change and the second particular change, ceasing modifying the first virtual object by locking a set of dimensions of the additional augmented reality content at a current representation in the first scene.
4 . The method of claim 3 , wherein ceasing modifying the first virtual object comprises ceasing generating the additional augmented reality content in the first scene to include an additional portion of augmented reality content.
5 . The method of claim 3 , further comprising:
detecting from a third set of frames, a second gesture, the second gesture corresponding to a second pinch gesture;
detecting a fifth location and a fifth position of a first representation of the first finger from the second gesture and a sixth location and a sixth position of the second representation of the second finger from the second gesture; and
detecting a second collision event corresponding to a fourth collider of the first representation of the first finger and a fifth collider of the second representation of the second finger intersecting with a sixth collider of the first virtual object, the second collision event being at a different portion of the first virtual object than the first collision event.
6 . The method of claim 5 , further comprising:
detecting a first respective change in the first location and the first position and a second respective change in the sixth location and the sixth position; and
in response to the first respective change and the second respective change, modifying the first virtual object to include a second additional augmented reality content based at least in part on the first respective change and the second respective change, the second additional augmented reality content being different than the additional augmented reality content.
7 . The method of claim 6 , further comprising:
rendering the first virtual object based at least in part on the modifying to include the second additional augmented reality content within the first scene; and
providing for display the rendered first virtual object within the first scene.
8 . The method of claim 1 , wherein modifying the first virtual object to include the additional augmented reality content based at least in part on the first change and the second change comprises:
generating the additional augmented reality content as attached to the first virtual object, the additional augmented reality content comprising at least one dimension from a first set of dimensions that are smaller than a second set of dimensions of the first virtual object.
9 . The method of claim 8 , wherein the at least one dimension comprises a length, width, or depth of the additional reality content.
10 . A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
detecting, from a set of frames, a first gesture, the first gesture corresponding to a pinch gesture;
detecting a first location and a first position of a first representation of a first finger from the first gesture and a second location and a second position of a second representation of a second finger from the first gesture, the first location comprising a first set of spatial coordinates of the first representation of the first finger in three dimensional (3D) space, the second location comprising a second set of spatial coordinates of the second representation of the second finger in the 3D space, the first position comprising a first orientation or first pose of the first representation of the first finger, and the second position comprising a second orientation or second pose of the second representation of the second finger;
detecting a first collision event corresponding to a first collider of the first representation of the first finger and a second collider of the second representation of the second finger intersecting with a third collider of a first virtual object to initiate selection of a first portion of the first virtual object as a first starting point for generating first additional augmented reality content that extends from the first starting point;
detecting a first change in the first location and the first position and a second change in the second location and the second position;
in response to the first change and the second change, modifying the first virtual object to include an additional augmented reality content based at least in part on the first change and the second change;
rendering the first virtual object based at least in part on the modifying within a first scene;
providing for display the rendered first virtual object within the first scene,
wherein the rendered first virtual object is provided for display within a messaging application executing on a first client device; and
transmitting, via the messaging application, the rendered first virtual object to a second client device of a recipient user of the messaging application.
11 . The system of claim 10 , wherein the first change and the second change does not result in an increase in a distance greater than a threshold value between the first representation of the first finger and the second representation of the second finger, and the first finger and the second finger is from a same hand.
12 . The system of claim 10 , wherein the operations further comprise:
detecting, from a second set of frames, a third location and a third position of the first representation of the first finger and a fourth location and a fourth position of the second representation of the second finger;
detecting a first particular change in the third location and the third position and a second particular change in the fourth location and the fourth position, wherein the first particular change and the second particular change results in an increase in a distance greater than a threshold value between the first representation of the first finger and the second representation of the second finger, and the first finger and the second finger is from a same hand; and
in response to detecting the first particular change and the second particular change, ceasing modifying the first virtual object by locking a set of dimensions of the additional augmented reality content at a current representation in the first scene.
13 . The system of claim 12 , wherein ceasing modifying the first virtual object comprises ceasing generating the additional augmented reality content in the first scene to include an additional portion of augmented reality content.
14 . The system of claim 12 , wherein the operations further comprise:
detecting, from a third set of frames, a second gesture, the second gesture corresponding to a second pinch gesture;
detecting a fifth location and a fifth position of a first representation of the first finger from the second gesture and a sixth location and a sixth position of the second representation of the second finger from the second gesture; and
detecting a second collision event corresponding to a fourth collider of the first representation of the first finger and a fifth collider of the second representation of the second finger intersecting with a sixth collider of the first virtual object, the second collision event being at a different portion of the first virtual object than the first collision event.
15 . The system of claim 14 , wherein the operations further comprise:
detecting a first respective change in the first location and the first position and a second respective change in the sixth location and the sixth position; and
in response to the first respective change and the second respective change, modifying the first virtual object to include a second additional augmented reality content based at least in part on the first respective change and the second respective change, the second additional augmented reality content being different than the additional augmented reality content.
16 . The system of claim 15 , wherein the operations further comprise:
rendering the first virtual object based at least in part on the modifying to include the second additional augmented reality content within the first scene; and
providing for display the rendered first virtual object within the first scene.
17 . The system of claim 10 , wherein modifying the first virtual object to include the additional augmented reality content based at least in part on the first change and the second change comprises:
generating the additional augmented reality content as attached to the first virtual object, the additional augmented reality content comprising at least one dimension from a first set of dimensions that are smaller than a second set of dimensions of the first virtual object.
18 . The system of claim 17 , wherein the at least one dimension comprises a length, width, or depth of the additional reality content.
19 . The method of claim 1 , further comprising:
detecting a second collision event corresponding to the first collider and the second collider intersecting with the third collider at a different portion of the first virtual object to initiate selection of a second portion of the first virtual object as a second starting point for generating second additional augmented reality content that extends from the second starting point, wherein the second additional augmented reality content is different from the first additional augmented reality content.
20 . The method of claim 1 , wherein the messaging application is configured for at least one of: starting or stopping recording in response to a second gesture; automatically setting a cut point for trimming video; restricting cursor movement to at least one axis in response to a third gesture; performing ray casting and animating virtual objects along a determined path in response to a fourth gesture; displaying a color palette in response to a fifth gesture; rotating, expanding, or reducing a size of a virtual object in response to a sixth gesture; rendering additional menu items in response to a seventh gesture; fixing positions of menu items in response to an eighth gesture; or removing menu items in response to a ninth gesture.
21 . The method of claim 20 , wherein:
the second gesture comprises an open hand with a palm raised and showing,
the third gesture comprises a finger pointing gesture for cursor control,
the fourth gesture comprises a firearm-like hand sign with a thumb and an index finger,
the fifth gesture comprises a pinch gesture for selecting from a color palette,
the sixth gesture comprises both hands positioned to grasp a virtual object,
the seventh gesture comprises both hands moving outward toward edges of an interface,
the eighth gesture comprises an open hand with a thumb spread away from other fingers, and
the ninth gesture comprises both hands moving inward toward a center of an interface.
22 . A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
detecting, from a set of frames, a first gesture, the first gesture corresponding to a pinch gesture;
detecting a first location and a first position of a first representation of a first finger from the first gesture and a second location and a second position of a second representation of a second finger from the first gesture, the first location comprising a first set of spatial coordinates of the first representation of the first finger in three dimensional (3D) space, the second location comprising a second set of spatial coordinates of the second representation of the second finger in the 3D space, the first position comprising a first orientation or first pose of the first representation of the first finger, and the second position comprising a second orientation or second pose of the second representation of the second finger;
detecting a first collision event corresponding to a first collider of the first representation of the first finger and a second collider of the second representation of the second finger intersecting with a third collider of a first virtual object to initiate selection of a first portion of the first virtual object as a first starting point for generating first additional augmented reality content that extends from the first starting point;
detecting a first change in the first location and the first position and a second change in the second location and the second position;
in response to the first change and the second change, modifying the first virtual object to include an additional augmented reality content based at least in part on the first change and the second change;
rendering the first virtual object based at least in part on the modifying within a first scene;
providing for display the rendered first virtual object within the first scene,
wherein the rendered first virtual object is provided for display within a messaging application executing on a first client device; and
transmitting, via the messaging application, the rendered first virtual object to a second client device of a recipient user of the messaging application.