Granted Patent E1

US 50,598 · App. 18/095,946 · Granted Sep 23, 2025

Inventors: Jonathan Ravasz (London, GB); Jasper Stevens (London, GB); Adam Tibor Varga (London, GB); Etienne Pinchon (London, GB); Simon Charles Tickner (Canterbury, GB); Jennifer Lynn Spurlock (Seattle, WA); Kyle Eric Sorge-Toomey (Seattle, WA); Robert Ellis (London, GB); Barrett Fox (Berkeley, CA)

Assignee: Meta Platforms Technologies, LLC

G06F3/017G06F3/011G06F3/012G06F3/0304G06F3/04815G06F3/0482G06F3/04842G06F3/04847G06T19/006

Abstract

Claims (145)

1. An artificial reality system comprising:

an image capture device configured to capture image data;

a head mounted display (HMD) configured to output artificial reality content;

a gesture detector comprising processing circuitry configured to identify, from the image data, a menu activation gesture comprising a configuration of a hand in a substantially upturned orientation of the hand and a pinching configuration of a thumb and a finger of the hand and identify, from the image data and subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand;

a user interface (UI) engine configured to, in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface, and in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and

a rendering engine configured to render the artificial reality content, the menu interface, and the translation of the slidably engageable UI element from the first position relative to the user menu interface to the second position relative to the user menu interface for display at the HMD.

2. The artificial reality system of claim 1 ,

wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and

wherein the UI engine is configured to highlight one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

3. The artificial reality system of claim 2 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.

4. The artificial reality system of claim 1 ,

wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface,

wherein to translate the slidably engageable UI element to the second position relative to the menu interface, the UI engine is configured to slide the slidably engageable UI element along the dimension of the menu interface to the second position relative to the menu interface.

5. The artificial reality system of claim 1 ,

wherein the menu sliding gesture comprises motion of the hand in a substantially first direction,

wherein the gesture detector is configured to identify, from the image data, motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction,

wherein the UI engine is further configured to, in response to the motion of the hand in the substantially second direction, translate the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

6. The artificial reality system of claim 1 , further comprising:

an application engine comprising processing circuitry for execution of one or more artificial reality applications,

wherein the gesture detector is configured to identify, from the image data, a selection gesture subsequent to the menu sliding gesture, and

wherein the application engine is configured to perform an action in response to the selection gesture.

7. The artificial reality system of claim 6 , wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to no longer be in the pinching configuration.

8. The artificial reality system of claim 1 ,

wherein the gesture detector is further configured to identify, from the image data, a menu prompt gesture prior to the menu activation gesture, and

wherein the UI engine is further configured to generate a menu prompt element in response to the menu prompt gesture.

9. The artificial reality system of claim 8 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger, and

wherein the UI engine generates the menu prompt element in the space between a thumb and a finger of a virtual hand.

10. The artificial reality system of claim 9 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.

11. The artificial reality system of claim 1 , wherein the image capture device is integrated with the HMD.

12. A method comprising:

obtaining, by an artificial reality system including a head mounted display (HMD), image data via an image capture device;

identifying, by the artificial reality system from the image data, a menu activation gesture, the menu activation gesture comprising a configuration of a hand in a substantially upturned orientation of the hand and a pinching configuration of a thumb and a finger of the hand;

generating, by the artificial reality system in response to the menu activation gesture, a menu interface and a slidably engageable UI element at a first position relative to the menu interface;

identifying, from the image data and subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand;

in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface, by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and

rendering, by the artificial reality system, artificial reality content, the menu interface, and the translation of the slidably engageable UI element from the first position relative to the menu interface to the second position relative to the user menu interface for display at the HMD.

13. The method of claim 12 , wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, the method further comprising:

highlighting, by the artificial reality system, one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

14. The method of claim 12 , wherein translating the slidably engageable UI element to the second position relative to the menu interface comprises sliding the slidably engageable UI element along a dimension of the menu interface to the second position relative to the menu interface.

15. The method of claim 12 ,

wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, the method further comprising:

identifying, by the artificial reality system from the image data, motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction; and

translating, by the artificial reality system in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

16. The method of claim 12 , further comprising:

identifying, by the artificial reality system from the image data, a menu prompt gesture; and

generating, by the artificial reality system, a menu prompt element in response to the menu prompt gesture.

17. A non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system to:

capture image data via an image capture device;

identify, from the image data, a menu activation gesture comprising a configuration of the a hand;

in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface;

identify, subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand;

in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and

render artificial reality content, the menu interface, and the slidably engageable UI element for display at a head mounted display (HMD).

18. The non-transitory, computer-readable medium of claim 17 , wherein the instructions further cause the one or more processors to:

identify, from the image data, a menu prompt gesture comprising the hand configured in a substantially upturned position with a space between a thumb and a finger; and

generate, in the space between a thumb and a finger of a virtual hand, a menu prompt element in response to the menu prompt gesture.

19. The non-transitory, computer-readable medium of claim 18 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.

20. The non-transitory, computer-readable medium of claim 17 , wherein the menu interface comprises one or more menu items, and wherein the instructions further cause the one or more processors to:

modify the appearance of one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

21. The non-transitory, computer-readable medium of claim 20 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.

22. The non-transitory, computer-readable medium of claim 17 ,

wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and

wherein the slidably engageable UI element is rendered translating along the dimension of the menu interface from the first position relative to the menu interface to the second position relative to the menu interface.

23. The non-transitory, computer-readable medium of claim 22 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.

24. The non-transitory, computer-readable medium of claim 17 , wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, and wherein the instructions further cause the one or more processors to:

identify, subsequent to the menu activation gesture, motion of the hand in a substantially second direction, the substantially second direction being substantially orthogonal to the substantially first direction; and

translate, in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

25. The non-transitory, computer-readable medium of claim 17 , wherein the instructions further cause the one or more processors to:

identify, subsequent to the menu sliding gesture, a selection gesture; and

perform an action in response to the selection gesture.

26. The non-transitory, computer-readable medium of claim 25 , wherein the menu interface comprises one or more menu items, the second position relates to a first menu item of the one or more menu items, and the performed action is associated with the first menu item.

27. The non-transitory, computer-readable medium of claim 25 ,

wherein the configuration of the hand of the menu activation gesture comprises the hand in a substantially upturned orientation and a pinching configuration of a thumb and a finger of the hand; and

wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to not be in a pinching configuration.

28. The non-transitory, computer-readable medium of claim 17 , wherein the instructions further cause the one or more processors to:

render a virtual hand for display at the HMD.

29. The non-transitory, computer-readable medium of claim 28 , wherein the virtual hand comprises one or more digits.

30. A method comprising:

capturing image data via an image capture device;

identifying, from the image data, a menu activation gesture comprising a configuration of a hand;

in response to the menu activation gesture, generating a menu interface and a slidably engageable UI element at a first position relative to the menu interface;

identifying, subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand; and

in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translating the slidably engageable UI element to a second position relative to the menu interface, and rendering the menu interface and the slidably engageable UI element for display at a head mounted display (HMD),

wherein the slidably engageable UI element is translated to the second position by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element.

31. The method of claim 30 , further comprising:

identifying a menu prompt gesture; and

generating a menu prompt element in response to the menu prompt gesture.

32. The method of claim 31 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger.

33. The method of claim 31 , wherein the menu prompt element is generated in the space between a thumb and a finger of a virtual hand.

34. The method of claim 33 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.

35. The method of claim 30 , wherein the menu interface comprises one or more menu items, the method further comprising:

modifying the appearance of one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

36. The method of claim 35 , wherein the one or more menu items correspond to respective applications executing on an artificial reality system.

37. The method of claim 30 ,

wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and

wherein translating the slidably engageable UI element to the second position relative to the menu interface comprises translating the slidably engageable UI element along the dimension of the menu interface from the first position relative to the menu interface to the second position relative to the menu interface.

38. The method of claim 37 , wherein the one or more menu items correspond to respective applications executing on an artificial reality system.

39. The method of claim 30 , wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, the method further comprising:

identifying, subsequent to the menu activation gesture, motion of the hand in a substantially second direction, the substantially second direction being substantially orthogonal to the substantially first direction; and

translating, in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

40. The method of claim 30 , further comprising:

identifying, subsequent to the menu sliding gesture, a selection gesture; and

performing an action in response to the selection gesture.

41. The method of claim 40 , wherein the menu interface comprises one or more menu items, the second position relates to a first menu item of the one or more menu items, and the performed action is associated with the first menu item.

42. The method of claim 40 ,

wherein the configuration of the hand of the menu activation gesture comprises the hand in a substantially upturned orientation and a pinching configuration of a thumb and a finger of the hand; and

wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring a thumb and a finger of the hand to not be in a pinching configuration.

43. The method of claim 30 , further comprising:

rendering a virtual hand for display at the HMD.

44. The method of claim 43 , wherein the virtual hand comprises one or more digits.

45. An artificial reality system comprising:

an image capture device configured to capture image data;

a head mounted display (HMD) configured to output artificial reality content;

a gesture detector comprising processing circuitry configured to identify, from the image data, a menu activation gesture comprising a configuration of a hand and identify, subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand,

a user interface (UI) engine comprising processing circuitry configured to, in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface, and in response to the menu sliding gesture that, via at least the configuration of the hand, engages the slidably engageable UI element, translate the slidably engageable UI element to a second position relative to the menu interface by sliding the slidably engageable UI element such that each motion of the hand, in at least one dimension, is translated to a motion of the menu sliding gesture that causes the sliding of the slidably engageable UI element; and

a rendering engine comprising processing circuitry configured to render the menu interface and the slidably engageable UI element for display at the HMD.

46. The artificial reality system of claim 45 ,

wherein the gesture detector is further configured to identify a menu prompt gesture, and

wherein the UI engine is further configured to generate a menu prompt element in response to the menu prompt gesture.

47. The artificial reality system of claim 46 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger.

48. The artificial reality system of claim 46 , wherein the UI engine generates the menu prompt element in the space between a thumb and a finger of a virtual hand.

49. The artificial reality system of claim 48 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.

50. The artificial reality system of claim 45 ,

wherein the menu interface comprises one or more menu items, and

wherein the UI engine is configured to modify the appearance of one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

51. The artificial reality system of claim 50 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.

52. The artificial reality system of claim 45 ,

wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface,

wherein to translate the slidably engageable UI element to the second position relative to the menu interface, the UI engine is configured to translate the slidably engageable UI element along the dimension of the menu interface from the first position relative to the menu interface to the second position relative to the menu interface.

53. The artificial reality system of claim 52 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.

54. The artificial reality system of claim 45 ,

wherein the menu sliding gesture comprises motion of the hand in a substantially first direction,

wherein the gesture detector is configured to identify motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction, and

wherein the UI engine is further configured to, in response to the motion of the hand in the substantially second direction, translate the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

55. The artificial reality system of claim 45 , further comprising:

an application engine comprising processing circuitry for execution of one or more artificial reality applications,

wherein the gesture detector is configured to identify a selection gesture subsequent to the menu sliding gesture, and

wherein the application engine is configured to perform an action in response to the selection gesture.

56. The artificial reality system of claim 55 , wherein the menu interface comprises one or more menu items, the second position relates to a first menu item of the one or more menu items, and the performed action is associated with the first menu item.

57. The artificial reality system of claim 55 ,

wherein the configuration of the hand of the menu activation gesture comprises the hand in a substantially upturned orientation and a pinching configuration of a thumb and a finger of the hand; and

wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to not be in a pinching configuration.

58. The artificial reality system of claim 45 , wherein the image capture device is integrated with the HMD.

59. The artificial reality system of claim 45 , wherein the rendering engine is further configured to render a virtual hand for display at the HMD.

60. The artificial reality system of claim 59 , wherein the virtual hand comprises one or more digits.

Continuity (1)

References Cited (239)