Granted Patent B1

US 10,976,804 · App. 16/506,183 · Granted Apr 13, 2021

Inventors: Charlene Mary Atlas (Redmond, WA); Chad Austin Bramwell (Sammamish, WA); Mark Terrano (Redmond, WA); Caryn Vainio (Hobart, WA); Xiaoyu Liu (Kirkland, WA)

Assignee: Facebook Technologies, LLC

G06F3/011G02B27/017G06F3/017G06F3/04815G06T19/006

Abstract

Claims (92)

1. An artificial reality system comprising:

an image capture device configured to capture image data;

a head-mounted display (HMD) configured to output artificial reality content;

a memory; and

processing circuitry in communication with the memory, the processing circuitry being configured to implement:

a virtual surface application configured to detect a physical peripheral device from the image data;

a gesture detector configured to detect a pose of the peripheral device,

wherein the virtual surface application is configured to:

generate, based on the pose of the peripheral device, a virtual pointer along at least a portion of a line between the peripheral device and a virtual surface, the virtual pointer pointing at a location of the virtual surface; and

perform one or more actions based on the location of the virtual surface; and

a rendering engine configured to render the virtual pointer and the virtual surface for display at the HMD.

2. The artificial reality system of claim 1 ,

wherein the gesture detector is configured to detect a pointer drawing gesture in which the user modifies a pose of the peripheral device to cause the virtual pointer to point at multiple locations of the virtual surface at different times,

wherein to perform the one or more actions, the virtual surface application is configured to generate, in response to the peripheral device drawing gesture, one or more virtual markings at the locations of the virtual surface, and

wherein the rendering engine is configured to render the one or more virtual markings for display at the HMD.

3. The artificial reality system of claim 2 ,

wherein the virtual surface application is configured to generate a user interface comprising a draw element,

wherein the pointer drawing gesture comprises selection of the draw element, and

wherein the rendering engine is configured to render, at a user interface position locked relative to a position of the peripheral device, the user interface for display at the HMD.

4. The artificial reality system of claim 1 ,

wherein, to generate the virtual pointer, the virtual surface application is configured to generate a visible element, and

wherein, to render the virtual pointer, the rendering engine is configured to render the visible element at the location of the virtual surface.

5. The artificial reality system of claim 1 ,

wherein, to generate the virtual pointer, the virtual surface application is configured to generate one or more visible elements along at least a portion of the line between the peripheral device and the location of the virtual surface.

6. The artificial reality system of claim 4 , wherein the one or more visible elements are partially transparent.

7. The artificial reality system of claim 1 ,

wherein the virtual surface application is configured to generate a user interface comprising one or more modify elements that allow a user to change a color or width of the virtual pointer,

wherein the virtual surface application is configured to change the color or width of the virtual pointer when the one or more modify elements are selected by a user, and

wherein the rendering engine is configured to render, at a user interface position locked relative to a position of the peripheral device, the user interface for display at the HMD.

8. The artificial reality system of claim 1 ,

wherein the gesture detector is configured to detect a peripheral device gesture performed by a user with respect to the peripheral device,

wherein the virtual surface includes one or more virtual content items and the location of the virtual surface corresponds to a selected virtual content item of the one or more virtual content items,

wherein the peripheral device gesture comprises a virtual content item transformation gesture comprising a translation or rotation movement of the peripheral device in physical space, and

wherein to perform the one or more actions, the virtual surface application is configured to, in response to the virtual content item transformation gesture, scale, rotate, or translate the selected virtual content item, relative to the virtual surface, commensurate with the translation or rotation movement of the peripheral device in physical space.

9. The artificial reality system of claim 8 ,

wherein the virtual surface application is configured to generate a user interface comprising a virtual content transformation element,

wherein virtual content item transformation gesture comprises selecting the virtual content transformation element, and

wherein the rendering engine is configured to render, at a user interface position locked relative to a position of the peripheral device, the user interface for display at the HMD.

10. The artificial reality system of claim 1 ,

wherein the gesture detector is configured to detect a peripheral device gesture performed by a user with respect to the peripheral device,

wherein the virtual surface includes one or more virtual content items and the location of the virtual surface corresponds to a selected virtual content item of the one or more virtual content items,

wherein the peripheral device gesture comprises an annotation gesture, and

wherein to perform the one or more actions, the virtual surface application is configured to, in response to the annotation gesture, generate the virtual surface to annotate the selected virtual content item.

11. A method comprising:

obtaining, by an artificial reality system including a head mounted display (HMD), image data via an image capture device, the HMD configured to output artificial reality content;

detecting, by the artificial reality system, a physical peripheral device from the image data;

detecting, by the artificial reality system, a pose of the peripheral device;

generating, by the artificial reality system, based on the pose of the peripheral device, a virtual pointer along at least a portion of a line between the peripheral device and a virtual surface, the virtual pointer pointing at a location of the virtual surface;

performing, by the artificial reality system, one or more actions based on the location of the virtual surface; and

rendering, by the artificial reality system, the virtual pointer and the virtual surface for display at the HMD.

12. The method of claim 11 , further comprising:

detecting, by the artificial reality system, a pointer drawing gesture in which the user modifies a pose of the peripheral device to cause the virtual pointer to point at multiple locations of the virtual surface at different times,

wherein performing, by the artificial reality system, the one or more actions comprises generating one or more virtual markings at the locations of the virtual surface; and

rendering, by the artificial reality system, the virtual markings for display at the HMD.

13. The method of claim 12 , further comprising:

generating, by the artificial reality system, a user interface comprising a drawing element,

wherein the pointer drawing gesture comprises selection of the drawing element; and

rendering, by the artificial reality system at a user interface position locked relative to a position of the peripheral device, the user interface for display at the HMD.

14. The method of claim 11 , further comprising:

generating, by the artificial reality system, a user interface comprising one or more modify elements that allow a user to change a color or width of the virtual pointer;

changing, by the artificial reality system, the color or width of the virtual pointer when the one or more modify elements are selected by a user; and

rendering, by the artificial reality system at a user interface position locked relative to a position of the peripheral device, the user interface for display at the HMD.

15. The method of claim 11 , further comprising:

detecting, by the artificial reality system, a peripheral device gesture performed by a user with respect to the peripheral device,

wherein the virtual surface includes one or more virtual content items and the location of the virtual surface corresponds to a selected virtual content item of the one or more virtual content items,

wherein the peripheral device gesture comprises a virtual content item transformation gesture comprising a translation or rotation movement of the peripheral device in physical space; and

in response to the virtual content item transformation gesture, scaling, rotating, or translating, by the artificial reality system, the selected virtual content item, relative to the virtual surface, commensurate with the translation or rotation movement of the peripheral device in physical space.

16. The method of claim 11 , wherein:

detecting, by the artificial reality system, a peripheral device gesture performed by a user with respect to the peripheral device,

wherein the virtual surface includes one or more virtual content items and the location of the virtual surface corresponds to a selected virtual content item of the one or more virtual content items,

wherein the peripheral device gesture comprises an annotation gesture, and

generating, by the artificial reality system in response to the virtual content item annotation gesture, the virtual surface to annotate the selected virtual content item.

17. A non-transitory, computer-readable medium comprising instructions that, when executed, cause processing circuitry of an artificial reality system to:

obtain image data via an image capture device;

detect a physical peripheral device from the image data;

detect a pose of the peripheral device;

generate, based on the pose of the peripheral device, a virtual pointer along at least a portion of a line between the peripheral device and to a virtual surface, the virtual pointer pointing a location of the virtual surface;

perform one or more actions based on the location of the virtual surface; and

render the virtual pointer and the virtual surface for display at the HMD.

18. The non-transitory, computer-readable medium of claim 17 , wherein the instructions further cause the processing circuitry to:

detect a pointer drawing gesture in which the user modifies a pose of the peripheral device to cause the virtual pointer to point at multiple locations of the virtual surface at different times,

wherein performing the one or more actions comprises generating, in response to the peripheral device drawing gesture, one or more virtual markings at the locations of the virtual surface, and

render the one or more virtual markings for display at the HMD.

19. The non-transitory, computer-readable medium of claim 18 , wherein the instructions further cause the processing circuitry to:

generate a user interface comprising a drawing element,

wherein the pointer drawing gesture comprises selection of the drawing element; and

render, at a user interface position locked relative to a position of the peripheral device, the user interface for display at the HMD.

20. The non-transitory, computer-readable medium of claim 17 , wherein the instructions further cause the processing circuitry to:

detect a peripheral device gesture performed by a user with respect to the peripheral device,

wherein the virtual surface includes one or more virtual content items and the location of the virtual surface corresponds to a selected virtual content item of the one or more virtual content items,

wherein the peripheral device gesture comprises a virtual content item transformation gesture comprising a translation or rotation movement of the peripheral device in physical space; and

in response to the virtual content item transformation gesture, scale, rotate, or translate the selected virtual content item, relative to the virtual surface, commensurate with the translation or rotation movement of the peripheral device in physical space.

Assignments (2)

Cited By (12)

US 12,217,347 US 12,223,104 US 12,242,666 US 12,259,855 US 12,315,363 US 12,373,024 US 12,379,786 US 12,456,084 US 12,561,005 US 12,639,885 US 12,646,272 US 12,681,587