Granted Patent B2

US 12,229,331 · App. 18/046,636 · Granted Feb 18, 2025

Inventors: Jeffrey Roger Powers (San Francisco, CA); Vikas Reddy (Boulder, CO); Yuping Lin (Boulder, CO)

Assignee: XRPro, LLC

G06F3/011G06F3/0346G06T7/13G06T7/181G06T7/50G06T7/73G06T7/90G06T19/006G06T19/20G06T2207/10024G06T2219/2004

Abstract

Claims (79)

1. A method comprising:

presenting, on a display of a virtual reality or mixed reality system, a virtual scene to a user;

detecting an edge of a physical object within a physical environment from one or more images captured by the virtual reality or mixed reality system, the physical object excluded from the virtual scene;

determining a six degree of freedom pose associated with a user;

determining the user is within a threshold distance from the physical object; and

determining a field of view of the user based at least in part on the six degree of freedom pose of the user;

determining the edge is not within the field of view; and

displaying the edge of the physical object and an indicator of the edge of the physical object within a field of view of the user within the virtual scene being presented to the user.

2. The method as recited in claim 1 , wherein detecting the edge of the physical object includes:

detecting a first line segment based on a first color gradient within the one or more images;

detecting a second line segment based on a second color gradient within the one or more images;

merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and

locating the edge within a three-dimensional model of the physical environment.

3. The method as recited in claim 2 , wherein detecting the edge of the physical object further comprises adjusting location of the edge within the three-dimensional model.

4. The method as recited in claim 1 , wherein detecting the edge of the physical object includes:

detecting a first edgelet from the one or more images;

detecting a second edgelet from the one or more images;

detecting a continuous gradient between the first edgelet and the second edgelet;

joining the first edgelet with the second edgelet into a joined edgelet based at least in part on the continues gradient;

estimating a contour based at least in part on the joined edgelet; and

determining a surface based at least in part on the contour; and

wherein the edge is an edge of the surface.

5. The method as recited in claim 1 , wherein the one or more images are captured by a monocular camera.

6. The method as recited in claim 1 , further comprising:

determining a direction of movement associated with the user based at least in part on inertial measurement unit data collected by a inertial measurement unit of the virtual reality or mixed reality system;

determining a collision risk between the user and the physical object based at least in part on a direction of movement of the user, the six degree of freedom pose of the user, and a location of the edge; and

wherein displaying the edge of the physical object within a virtual scene is based at least in part on the collision risk being greater than a threshold.

7. The method as recited in claim 1 , wherein the indicator of the edge within the field of view of the user is a line projected from the edge of the physical object.

8. The method as recited in claim 1 , wherein the indicator is at least one of a ray projecting from the edge or the edge into a field of view of the user.

9. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:

presenting, on a display of a virtual reality or mixed reality system, a virtual scene to a user;

detecting an edge of a physical object within a physical environment from one or more images captured by a virtual reality or mixed reality system, the physical object excluded from the virtual scene;

determining a six degree of freedom pose associated with a user;

determining, based at least in part on the six degree of freedom pose and an accumulated drift, that the user is within a threshold distance from the physical object; and

displaying the edge of the physical object within a virtual scene being presented to the user.

10. The one or more non-transitory computer-readable media as recited in claim 9 , wherein detecting the edge of the physical object includes:

detecting a first line segment based on a first color gradient within the one or more images;

detecting a second line segment based on a second color gradient within the one or more images;

merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and

locating the edge within a three-dimensional model of the physical environment.

11. The one or more non-transitory computer-readable media as recited in claim 10 , wherein detecting the edge of the physical object further comprises adjusting location of the edge within the three-dimensional model.

12. The one or more non-transitory computer-readable media as recited in claim 9 , wherein detecting the edge of the physical object includes:

detecting a first edgelet from the one or more images;

detecting a second edgelet from the one or more images;

detecting a continuous gradient between the first edgelet and the second edgelet;

joining the first edgelet with the second edgelet into a joined edgelet based at least in part on the continues gradient;

estimating a contour based at least in part on the joined edgelet; and

determining a surface based at least in part on the contour; and

wherein the edge is an edge of the surface.

13. The one or more non-transitory computer-readable media as recited in claim 9 , wherein the one or more images are captured by a monocular camera.

14. The one or more non-transitory computer-readable media as recited in claim 9 , the operations further comprising:

determining a direction of movement associated with the user based at least in part on inertial measurement unit data collected by a inertial measurement unit of the virtual reality or mixed reality system;

determining a collision risk between the user and the physical object based at least in part on a direction of movement of the user, the six degree of freedom pose of the user, and a location of the edge; and

wherein displaying the edge of the physical object within a virtual scene is based at least in part on the collision risk being greater than a threshold.

15. The one or more non-transitory computer-readable media as recited in claim 9 , wherein displaying the edge of the physical object within a virtual scene includes:

determining a field of view of the user based at least in part on the six degree of freedom pose of the user;

determining the edge is not within the field of view; and

displaying an indicator of the edge within a field of view of the user.

16. A system comprising:

one or more sensors for capturing one or more images of a physical environment;

one or more processors; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

detecting an edge of a physical object within the physical environment within the one or more images, the physical object excluded from the virtual scene;

determining a six degree of freedom pose associated with a user;

determining the user is within a threshold distance from the physical object;

determining a field of view of the user based at least in part on the six degree of freedom pose of the user;

determining the edge is not within the field of view; and

displaying the edge of the physical object within a virtual scene being presented to the user.

17. The method as recited in claim 16 , wherein detecting the edge of the physical object includes:

detecting a first line segment based on a first color gradient within the one or more images;

detecting a second line segment based on a second color gradient within the one or more images;

merging the first line segment to the second line segment into the edge based at least in part on a similarly of the first color gradient to the second color gradient; and

locating the edge within a three-dimensional model of the physical environment.

18. The method as recited in claim 16 , wherein the operations further comprise:

determining a direction of movement associated with the user based at least in part on inertial measurement unit data collected by a inertial measurement unit of the virtual reality or mixed reality system;

determining a collision risk between the user and the physical object based at least in part on a direction of movement of the user, the six degree of freedom pose of the user, and a location of the edge; and

wherein displaying the edge of the physical object within a virtual scene is based at least in part on the collision risk being greater than a threshold.

19. The method as recited in claim 16 , wherein the indicator is at least one of a ray projecting from the edge or the edge into a field of view of the user.

20. The method as recited in claim 16 , wherein the one or more sensors are one or more monocular cameras.

Assignments (2)

Continuity (5)

References Cited (30)

Cited By (1)

US 12,366,914