Granted Patent B2

US 12,223,104 · App. 17/498,284 · Granted Feb 11, 2025

Inventors: Michael James LeBeau (London, GB); John Nicholas Jitkoff (Seattle, WA)

Assignee: Meta Platforms Technologies, LLC

G06F3/014G06F3/017G06T11/00

Abstract

Claims (57)

1. A method for providing partial passthrough video to a user of a virtual reality device, the method comprising:

displaying, to the user, a computer generated virtual environment;

detecting a hand passthrough trigger event based on a user action and/or a condition that occurs in the user's real-world environment;

in response to detecting the hand passthrough trigger event, identifying a hand passthrough video feed, from one or more cameras of the virtual reality device, that captures the user's real-world environment;

aligning the hand passthrough video feed with a portion of the virtual environment by matching a location of a real-world object and/or a location of a real-world hand of the user to a corresponding location in the virtual environment; and

dynamically overlaying the hand passthrough video feed, comprising the real world hand of the user and/or the real-world object, over the aligned portion of the virtual environment, wherein the dynamically overlaying provides, to the user via the hand passthrough video, a depth accurate view of a) the real-world hand of the user and/or b) the real-world object.

2. The method of claim 1 , wherein the detecting the hand passthrough trigger event comprises identifying that the user made a particular gesture based on a monitored pose of the user's hand, wherein the pose of the user's hand is monitored A) using input from one or more sensors attached to the user's hand or wrist and/or B) by analyzing images captured by the virtual reality device.

3. The method of claim 1 , wherein the detecting the hand passthrough trigger event comprises: a) identifying that a hand of the user exited a designated area; or b) determining that a hand of the user is within a threshold distance of a real-world surface or object.

4. The method of claim 1 , wherein the detecting the hand passthrough trigger event comprises identifying a predicted collision between a part of the user and the real-world object by analyzing, with a determined current hand velocity and trajectory, a kinematic model of the user.

5. The method of claim 1 , wherein the detecting the hand passthrough trigger event comprises determining that the user spoke a particular voice command or pressed a particular button associated with the virtual reality device.

6. The method of claim 1 ,

wherein the detecting the hand passthrough trigger event is based on an action of a hand of the user; and

wherein the identifying the hand passthrough video feed comprises:

identifying, in one or more images from the one or more cameras of the virtual reality device, the hand of the user that performed the action; and

identifying an area, in the one or more images from the one or more cameras of the virtual reality device, around the identified hand.

7. The method of claim 1 , wherein the identifying the hand passthrough video feed comprises:

determining a point where a particular straight or curved line, extending from a hand of the user, intersects with the real-world object or real-world surface; and

identifying an area, in one or more images from the one or more cameras of the virtual reality device, around the determined point.

8. The method of claim 1 ,

wherein the detecting the hand passthrough trigger event is based on an identification of the real-world object in one or more images from the one or more cameras of the virtual reality device; and

wherein the identifying the hand passthrough video feed comprises identifying an area, in the one or more images from the one or more cameras of the virtual reality device, around the identified real-world object.

9. The method of claim 1 ,

wherein the detecting the hand passthrough trigger event is based on an identification of the real-world object or a surface in one or more images from the one or more cameras of the virtual reality device; and

wherein a size of the hand passthrough video feed is set based on a determined distance between a hand of the user and the identified real-world object or surface.

10. The method of claim 1 , wherein the identifying the hand passthrough video feed comprises:

identifying, in one or more images from the one or more cameras of the virtual reality device, a first hand of the user and a second hand of the user; and

identifying an area, in the one or more images from the one or more cameras of the virtual reality device, around the identified first hand, wherein the size of the area is based on a determined distance between the first hand and the second hand.

11. The method of claim 1 , wherein,

the dynamically overlaying the hand passthrough video feed comprises replacing the aligned portion of the virtual environment with the hand passthrough video feed, and

the dynamically overlaying the hand passthrough video feed simultaneously displays, to the user: a) a remaining portion of the virtual environment that is not covered by the overlaid passthrough video feed, and b) the user's real-world environment captured by the one or more cameras.

12. The method of claim 1 , wherein the dynamically overlaying is performed by:

tracking the user's hand in the user's real-world environment, wherein the alignment of the hand passthrough video feed with respect to the virtual environment is updated based on the tracking; and

moving the dynamically overlaid hand passthrough video feed based on the updated alignment to achieve the depth-accurate view.

13. The method of claim 1 , wherein the matching the location of the real-world object and/or the user's hand in the user's real-world environment to the corresponding location in the virtual environment includes matching coordinates associated with the handpassthrough video feed to coordinates in the virtual environment.

14. A computing system for providing partial passthrough video to a user of a virtual reality device, the computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

displaying, to the user, a computer generated virtual environment;

detecting a hand passthrough trigger event based on a user action and/or a condition that occurs in the user's real-world environment;

in response to detecting the hand passthrough trigger event, identifying a hand passthrough video feed, from one or more cameras of the virtual reality device, that captures the user's real-world environment;

aligning the hand passthrough video feed with a portion of the virtual environment by matching a location of a real-world object and/or a location of a real-world hand of the user to a corresponding location in the virtual environment; and

dynamically overlaying the hand passthrough video feed, comprising the real world hand of the user and/or the real-world object, over the aligned portion of the virtual environment, wherein the dynamically overlaying provides, to the user via the hand passthrough video, a depth accurate view of a) the real-world hand of the user and/or b) the real-world object.

15. The computing system of claim 14 , wherein the detecting the hand passthrough trigger event comprises determining that a hand of the user is within a threshold distance of the real-world object or a real-world surface.

16. The computing system of claim 14 , wherein the detecting the hand passthrough trigger event comprises identifying that the user made a particular gesture based on a monitored pose of the user's hand, wherein the pose of the user's hand is monitored A) using input from one or more sensors attached to the user's hand, wrist, or forearm and/or B) by analyzing images captured by the virtual reality device.

17. The computing system of claim 14 ,

wherein the detecting the hand passthrough trigger event is based on an action of a hand of the user; and

wherein the identifying the hand passthrough video feed comprises:

identifying, in one or more images from the one or more cameras of the virtual reality device, the hand of the user that performed the action; and

identifying an area, in the one or more images from the one or more cameras of the virtual reality device, around the identified hand.

18. The computing system of claim 14 , wherein the detecting the hand passthrough trigger event comprises identifying that a hand of the user exited a designated area.

19. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:

displaying, to the user, a computer generated virtual environment;

detecting a hand passthrough trigger event based on a user action and/or a condition that occurs in the user's real-world environment;

in response to detecting the hand passthrough trigger event, identifying a hand passthrough video feed, from one or more cameras of the virtual reality device, that captures the user's real-world environment;

aligning the hand passthrough video feed with a portion of the virtual environment presented to the user by the virtual reality device by matching a location of a real-world object and/or a location of a real-world hand of the user to a corresponding location in the virtual environment; and

dynamically overlaying the hand passthrough video feed, comprising the real world hand of the user and/or the real-world object, over the aligned portion of the virtual environment, wherein the dynamically overlaying provides, to the user via the hand passthrough video, a depth accurate view of a) the real-world hand of the user and/or b) the real-world object.

20. The computer-readable storage medium of claim 19 , wherein the identifying the hand passthrough video feed comprises:

determining a point where a particular straight or curved line, extending from a hand of the user, intersects with the real-world object or a real-world surface; and

identifying an area, in one or more images from the one or more cameras of the virtual reality device, around the determined point.

Assignments (2)

Continuity (2)

References Cited (258)

Cited By (1)

US 12,627,784