IP Library Granted Patent US 10,726,633
Granted Patent B2
US 10,726,633 · App. 15/858,275 · Granted Jul 28, 2020

Systems and methods for generating and displaying artificial environments based on real-world environments

Inventors: Eric Deng (Fremont, CA); Andrew Gold (Los Altos, CA)
Assignee: Facebook, Inc.
G06T19/20G06F3/011G06K9/00288G06T7/70G06T15/503G06T17/00G06T19/00G06T19/006H04N7/147H04N7/152H04N7/157G06T5/004G06T2219/024G06T2219/2021
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,726,633
App. No.
15/858,275
Granted
Jul 28, 2020
Kind
B2
Abstract

The disclosed computer-implemented method may include (1) identifying, via at least one sensor, an object of interest located within a real-world environment, (2) mapping an area of the real-world environment surrounding the object of interest, (3) generating a virtual environment based on the mapped area of the real-world environment surrounding the object of interest, and (4) displaying, in real-time, the object of interest within the virtual environment. Various other methods, systems, and computer-readable media are also disclosed.

Claims (67)

1. A computer-implemented method comprising:

identifying, via at least one sensor, an object of interest located within a real-world environment;

mapping an area of the real-world environment surrounding the object of interest by identifying an environmental object comprising at least one restricted access object within the real-world environment surrounding the object of interest, wherein identifying the environmental object comprises determining a location of the environmental object within the real-world environment;

generating a virtual environment based on the mapped area of the real-world environment surrounding the object of interest by generating a model for use in the virtual environment that represents the environmental object, the model comprising a security notification indicating that the restricted access object is subject to one or more security policies, the security policies indicating that the restricted access object is not to be displayed in the generated virtual environment; and

displaying, in real-time, the object of interest within the virtual environment, wherein displaying the object of interest within the virtual environment comprises displaying the model that represents the environmental object within the virtual environment at a location within the virtual environment that corresponds to the location of the environmental object in the real-world environment,

wherein displaying the model includes avoiding rendering the restricted access object within the displayed virtual environment, and

wherein displaying the model includes automatically pivoting from displaying the object of interest in the virtual environment to displaying the object of interest in a rich representation that includes real-time video upon determining that sufficient bandwidth is available to transfer the real-time video.

2. The computer-implemented method of claim 1 , wherein identifying the object of interest located within the real-world environment comprises determining a location of the object of interest within the real-world environment.

3. The computer-implemented method of claim 2 , wherein displaying the object of interest within the virtual environment further comprises:

determining a location within the virtual environment that corresponds to the determined location of the object of interest within the real-world environment; and

displaying the object of interest within the virtual environment at the determined location within the virtual environment that corresponds to the determined location of the object of interest within the real-world environment.

4. The computer-implemented method of claim 1 , wherein:

the object of interest comprises at least a portion of a person; and

displaying the object of interest within the virtual environment further comprises displaying the portion of the person within the virtual environment to an additional person.

5. The computer-implemented method of claim 4 , wherein identifying the object of interest comprises identifying the person based on at least one attribute of the person.

6. The computer-implemented method of claim 5 , wherein the attribute of the person comprises at least one of:

a biometric attribute of the person; or

an identifying token associated with the person.

7. The computer-implemented method of claim 5 , further comprising presenting a notification to the additional person comprising an indication of the identity of the person.

8. A computer-implemented method comprising:

identifying, via at least one sensor, an object of interest comprising at least one restricted access object located within a real-world environment;

mapping an area of the real-world environment surrounding the object of interest;

generating a virtual environment based on the mapped area of the real-world environment surrounding the object of interest; and

displaying, in real-time, the object of interest within the virtual environment by displaying a model that represents the restricted access object within the virtual environment, the model comprising:

a security notification indicating that the restricted access object is subject to one or more security policies, the security policies indicating that the restricted access object is not to be displayed in the generated virtual environment,

wherein displaying the model includes avoiding rendering the restricted access object within the displayed virtual environment, and

wherein displaying the model includes automatically pivoting from displaying the object of interest in the virtual environment to displaying the object of interest in a rich representation that includes real-time video upon determining that sufficient bandwidth is available to transfer the real-time video.

9. The computer-implemented method of claim 1 , wherein displaying the object of interest within the virtual environment further comprises displaying a real-time video image of the object of interest within the virtual environment.

10. The computer-implemented method of claim 1 , wherein displaying the object of interest within the virtual environment further comprises displaying the object of interest within the virtual environment via at least one of:

a video conferencing application;

an augmented reality environment; or

a virtual reality environment.

11. The computer-implemented method of claim 1 , wherein the sensor comprises a camera.

12. The computer-implemented method of claim 1 , wherein the sensor is included in at least one of a mobile robot or a telepresence robot located in the real-world environment.

13. A system comprising:

an identifying module, stored in memory, that identifies, via at least one sensor, an object of interest located within a real-world environment;

a mapping module, stored in memory, that maps an area of the real-world environment surrounding the object of interest by identifying an environmental object comprising at least one restricted access object within the real-world environment surrounding the object of interest, wherein identifying the environmental object comprises determining a location of the environmental object within the real-world environment;

a generating module, stored in memory, that generates a virtual environment based on the mapped area of the real-world environment surrounding the object of interest by generating a model for use in the virtual environment that represents the environmental object, the model comprising:

a security notification indicating that the restricted access object is subject to one or more security policies, the security policies indicating that the restricted access object is not to be displayed in the generated virtual environment;

a displaying module, stored in memory, that displays, in real-time, the object of interest within the virtual environment, wherein the displaying module displays the object of interest within the virtual environment by displaying the model that represents the environmental object within the virtual environment at a location within the virtual environment that corresponds to the location of the environmental object in the real-world environment,

wherein displaying the model includes avoiding rendering the restricted access object within the displayed virtual environment, and

wherein displaying the model includes automatically pivoting from displaying the object of interest in the virtual environment to displaying the object of interest in a rich representation that includes real-time video upon determining that sufficient bandwidth is available to transfer the real-time video; and

at least one physical processor that executes the identifying module, the mapping module, the generating module, and the displaying module.

14. The system of claim 13 , wherein the identifying module identifies the object of interest located within the real-world environment by determining a location of the object of interest within the real-world environment.

15. The system of claim 14 , wherein the displaying module further displays the object of interest within the virtual environment by:

determining a location within the virtual environment that corresponds to the determined location of the object of interest within the real-world environment; and

displaying the object of interest within the virtual environment at the determined location within the virtual environment that corresponds to the determined location of the object of interest within the real-world environment.

16. The system of claim 13 , wherein the displaying module further displays the object of interest within the virtual environment by displaying the object of interest within the virtual environment via at least one of:

a video conferencing application;

an augmented reality environment; or

a virtual reality environment.

17. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor of a computing system, cause the computing system to:

identify, via at least one sensor, an object of interest located within a real-world environment;

map an area of the real-world environment surrounding the object of interest by identifying an environmental object comprising at least one restricted access object within the real-world environment surrounding the object of interest, wherein identifying the environmental object comprises determining a location of the environmental object within the real-world environment;

generate a virtual environment based on the mapped area of the real-world environment surrounding the object of interest by generating a model for use in the virtual environment that represents the environmental object, the model comprising:

a security notification indicating that the restricted access object is subject to one or more security policies, the security policies indicating that the restricted access object is not to be displayed in the generated virtual environment;

display, in real-time, the object of interest within the virtual environment, wherein displaying the object of interest within the virtual environment comprises displaying the model that represents the environmental object within the virtual environment at a location within the virtual environment that corresponds to the location of the environmental object in the real-world environment,

wherein displaying the model includes avoiding rendering the restricted access object within the displayed virtual environment, and

wherein displaying the model includes automatically pivoting from displaying the object of interest in the virtual environment to displaying the object of interest in a rich representation that includes real-time video upon determining that sufficient bandwidth is available to transfer the real-time video.

18. The computer-implemented method of claim 1 , wherein generating the model for use in the virtual environment that represents the environmental object comprises:

obtaining, via the sensor, data associated with at least one attribute of the environmental object;

determining the attribute of the environmental object based on the data associated with the attribute of the environmental object; and

generating the model for use in the virtual environment such that the model shares the attribute with the environmental object.

19. The computer-implemented method of claim 8 , wherein identifying the object of interest located within the real-world environment comprises determining a location of the object of interest within the real-world environment.

20. The computer-implemented method of claim 19 , wherein displaying the object of interest within the virtual environment further comprises:

determining a location within the virtual environment that corresponds to the determined location of the object of interest within the real-world environment.

21. The computer-implemented method of claim 19 , wherein displaying the object of interest within the virtual environment at the determined location within the virtual environment that corresponds to the determined location of the object of interest within the real-world environment.

Assignments (2)
CHANGE OF NAME Recorded Jan 27, 2022
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058871/0336 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 4, 2018
From: DENG, ERIC; GOLD, ANDREW
To: FACEBOOK, INC.
Reel/Frame 044540/0601 →
Continuity (1)
Related Publication 20190206141A1 · Jul 4, 2019