IP Library Granted Patent US 11,113,891
Granted Patent B2
US 11,113,891 · App. 16/773,770 · Granted Sep 7, 2021

Systems, methods, and media for displaying real-time visualization of physical environment in artificial reality

Inventors: Gioacchino Noris (Zurich, CH); Jeng-Weei Lin (Redmond, WA)
Assignee: Facebook Technologies, LLC
G06T19/006G06T19/20G06T2219/2012
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,113,891
App. No.
16/773,770
Granted
Sep 7, 2021
Kind
B2
Abstract

Particular embodiments are directed to a passthrough feature. A computing system may display a virtual-reality scene on a device worn by a user. The system may receive a request to display a visual representation of at least a portion of a physical environment surrounding the user. The system may access data associated with the physical environment captured by camera(s) of the device. The system may generate, based the data, depth measurements of one or more objects in the physical environment. The system may generate, based on the depth measurements, one or more models of the one or more objects in the physical environment. The system may render an image based on a viewpoint of the user and the one or more models and, based on the image, generate the visual representation requested by the user. The visual representation may then be displayed with the virtual-reality scene to the user.

Claims (52)

1. A method comprising, by a computing system:

displaying a virtual-reality scene on a device worn by a user;

receiving a request from the user to display a visual representation of at least a portion of a physical environment surrounding the user;

accessing data associated with the physical environment captured by one or more cameras of the device;

generating, based on the data, depth measurements of one or more objects in the physical environment;

generating, based on the depth measurements, one or more three-dimensional (3D) models of the one or more objects in the physical environment;

rendering an output image based on a current viewpoint of an eye of the user and the one or more 3D models of the one or more objects in the physical environment;

generating, based on the output image, the visual representation requested by the user; and

displaying the visual representation with the virtual-reality scene to the user.

2. The method of claim 1 , wherein the visual representation is an outline of the one or more objects in the physical environment.

3. The method of claim 2 , wherein the displayed outline of the one or more objects is superimposed over the virtual-reality scene.

4. The method of claim 2 , wherein the displayed outline has a uniform color tint.

5. The method of claim 4 , wherein the virtual-reality scene is generated by an application and the uniform color tint is selected by the application.

6. The method of claim 1 , wherein generating the visual representation comprises applying a gradient filter to the output image.

7. The method of claim 1 , wherein the visual representation is generated by an operating system of the computing system.

8. The method of claim 1 , wherein the device worn by the user is a head-mounted device that blocks the user from seeing the physical environment directly.

9. The method of claim 1 , wherein:

the current viewpoint of the eye of the user corresponds to a first eye of the user; and

the visual representation and the virtual-reality scene are displayed on a first display of the device configured to be viewed by the first eye of the user.

10. The method of claim 9 , further comprising:

determining a second viewpoint of a second eye of the user;

rendering a second output image based on the second viewpoint of the user and the one or more 3D models of the objects;

generating a second visual representation of at least a portion of the physical environment surrounding the user; and

displaying the second visual representation with a second virtual-reality scene on a second display of the device configured to be viewed by the second eye of the user.

11. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

display a virtual-reality scene on a device worn by a user;

receive a request from the user to display a visual representation of at least a portion of a physical environment surrounding the user;

access data associated with the physical environment captured by one or more cameras of the device;

generate, based on the data, depth measurements of one or more objects in the physical environment;

generate, based on the depth measurements, one or more three-dimensional (3D) models of the one or more objects in the physical environment;

render an output image based on a current viewpoint of an eye of the user and the one or more 3D models of the one or more objects in the physical environment;

generate, based on the output image, the visual representation requested by the user; and

display the visual representation with the virtual-reality scene to the user.

12. The media of claim 11 , wherein the visual representation is an outline of the one or more objects in the physical environment.

13. The media of claim 12 , wherein the displayed outline of the one or more objects is superimposed over the virtual-reality scene.

14. The media of claim 12 , wherein the displayed outline has a uniform color tint.

15. The media of claim 14 , wherein the virtual-reality scene is generated by an application and the uniform color tint is selected by the application.

16. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

display a virtual-reality scene on a device worn by a user;

receive a request from the user to display a visual representation of at least a portion of a physical environment surrounding the user;

access data associated with the physical environment captured by one or more cameras of the device;

generate, based on the data, depth measurements of one or more objects in the physical environment;

generate, based on the depth measurements, one or more three-dimensional (3D) models of the one or more objects in the physical environment;

render an output image based on a current viewpoint of an eye of the user and the one or more 3D models of the one or more objects in the physical environment;

generate, based on the output image, the visual representation requested by the user; and

display the visual representation with the virtual-reality scene to the user.

17. The system of claim 16 , wherein the visual representation is an outline of the one or more objects in the physical environment.

18. The system of claim 17 , wherein the displayed outline of the one or more objects is superimposed over the virtual-reality scene.

19. The system of claim 17 , wherein the displayed outline has a uniform color tint.

20. The system of claim 19 , wherein the virtual-reality scene is generated by an application and the uniform color tint is selected by the application.

Assignments (2)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 18, 2020
From: NORIS, GIOACCHINO; LIN, JENG-WEEI
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 052153/0004 →
Cited By (6)
US 12,524,072 US 12,572,213 US 12,586,294 US 12,619,316 US 12,657,845 US 12,681,559