Granted Patent B2

US 11,334,212 · App. 16/435,116 · Granted May 17, 2022

Inventors: Jonathan Ravasz (London, GB); Jasper Stevens (London, GB); Adam Tibor Varga (London, GB); Etienne Pinchon (London, GB); Simon Charles Tickner (Canterbury, GB); Jennifer Lynn Spurlock (Seattle, WA); Kyle Eric Sorge-Toomey (Seattle, WA); Robert Ellis (London, GB); Barrett Fox (Berkeley, CA)

Assignee: Facebook Technologies, LLC

G06F3/04815G02B27/017G06F3/017G06F3/0482G06K9/00335

Abstract

Claims (62)

1. An artificial reality system comprising:

an image capture device configured to capture image data representative of a physical environment;

a head-mounted display (HMD) configured to output artificial reality content; and

processing circuitry having access to memory to implement:

a gesture detector configured to:

identify, from the captured image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration and a subsequent pulling motion towards the HMD while the two fingers are in the pinching configuration,

determine, from the captured image data, that the pinching configuration is in the vicinity of an initial position of a UI pinch element, and

identify, after identifying the gesture, further movement of the hand while the two fingers are in the pinching configuration,

a user interface (UI) engine configured to generate a UI input element in response to identifying the gesture associated with the UI pinch element,

a rendering engine configured to:

render the UI input element as a three-dimensional overlay to at least some of the artificial reality content in response to identifying the gesture associated with the UI pinch element, wherein the three-dimensional overlay has a size, and

modify the three-dimensional overlay by changing the size of the three-dimensional overlay in response to the further movement of the hand, and wherein the modifications to the three-dimensional overlay correspond to a distance between the hand and the initial position of the UI pinch element;

wherein the gesture detector is further configured to determine, from the captured image data, that the pinching configuration has ceased, and

wherein the rendering engine is further configured to, responsive to determining that the pinching configuration has ceased, remove the three-dimensional overlay of the UI input element from the artificial reality content.

2. The artificial reality system of claim 1 , wherein to modify the three-dimensional overlay, the rendering engine is further configured to:

further modify the size of the three-dimensional overlay to indicate a further adjusted value.

3. The artificial reality system of claim 2 , wherein the rendering engine is further configured to:

render the further modified three-dimensional overlay.

4. The artificial reality system of claim 1 , wherein the user interface engine is further configured to:

identify, responsive to determining that the pinching configuration has ceased, an input value; and

generate, based on the input value, updated artificial reality content.

5. The artificial reality system of claim 4 , wherein the rendering engine is further configured to:

render the updated artificial reality content.

6. The artificial reality system of claim 1 , wherein the HMD is further configured to output at least some of the captured image data representative of the physical environment.

7. The artificial reality system of claim 1 , wherein to identify the gesture, the gesture detector is further configured to:

identify the gesture as corresponding to an entry in a gesture library.

8. The artificial reality system of claim 1 , wherein the two fingers from the hand are a thumb and index finger.

9. The artificial reality system of claim 1 , wherein the subsequent pulling motion includes a pulling motion of the hand.

10. The artificial reality system of claim 1 , wherein the subsequent pulling motion includes a pulling motion of the two fingers.

11. The artificial reality system of claim 1 , wherein the image capture device is integrated within the HMD.

12. The artificial reality system of claim 1 , wherein the UI input element is at least one of:

a one-dimensional continuous variable slider;

a one-dimensional discrete variable slider;

a switch;

a menu;

a radial item picker;

a radial slider;

a color picker;

a volumetric color picker; or

a progress slider.

13. A method comprising:

capturing, by an image capture device, image data representative of a physical environment;

outputting, by a head-mounted display (HMD), artificial reality content;

identifying, from the captured image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration and a subsequent pulling motion towards the HMD while the two fingers are in the pinching configuration;

determining, from the captured image data, that the pinching configuration is in the vicinity of an initial position of a UI pinch element;

identifying, after identifying the gesture, further movement of the hand while the two fingers are in the pinching configuration;

generating a UI input element in response to identifying the gesture associated with the UI pinch element;

rendering the UI input element as a three-dimensional overlay to at least some of the artificial reality content in response to identifying the gesture associated with the UI pinch element, wherein the three-dimensional overlay has a size;

modify the three-dimensional overlay by changing the size of the three-dimensional overlay in response to the further movement of the hand, and wherein the modifications to the three-dimensional overlay correspond to a distance between the hand and the initial position of the UI pinch element;

determining, from the captured image data, that the pinching configuration has ceased; and

responsive to determining that the pinching configuration has ceased, removing at least some aspects of the three-dimensional overlay of the UI input element from the artificial reality content.

14. A non-transitory computer-readable medium comprising instructions for causing one or more processors of an artificial reality system to perform operations comprising:

capturing, by an image capture device, image data representative of a physical environment;

outputting, by a head-mounted display (HMD), artificial reality content;

identifying, from the captured image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration and a subsequent pulling motion towards the HMD while the two fingers are in the pinching configuration;

determining, from the captured image data, that the pinching configuration is in the vicinity of an initial position of a UI pinch element;

identifying, after identifying the gesture, further movement of the hand while the two fingers are in the pinching configuration;

generating a UI input element in response to identifying the gesture associated with the UI pinch element;

rendering the UI input element as a three-dimensional overlay to at least some of the artificial reality content in response to identifying the gesture associated with the UI pinch element, wherein the three-dimensional overlay has a size;

modify the three-dimensional overlay by changing the size of the three-dimensional overlay in response to the further movement of the hand, and wherein the modifications to the three-dimensional overlay correspond to a distance between the hand and the initial position of the UI pinch element;

determining, from the captured image data, that the pinching configuration has ceased; and

responsive to determining that the pinching configuration has ceased, removing at least some aspects of the three-dimensional overlay of the UI input element from the artificial reality content.

Assignments (2)

Continuity (1)

Cited By (17)

US 12,189,422 US 12,236,008 US 12,265,655 US 12,321,509 US 12,360,557 US 12,360,558 US 12,380,238 US 12,474,816 US 12,530,102 US 12,530,103 US 12,537,877 US 12,561,005 US 12,572,213 US 12,591,313 US 12,619,316 US 12,619,774 US 12,669,848