IP Library Granted Patent US 11,423,621
Granted Patent B1
US 11,423,621 · App. 16/880,207 · Granted Aug 23, 2022

Adaptive rendering in artificial reality environments

Inventors: Charlene Mary Atlas (Redmond, WA); Romain Bachy (Seattle, WA); Kevin James MacKenzie (Sammamish, WA); Nathan Matsuda (Seattle, WA); Thomas Scott Murdison (Kirkland, WA); Ocean Quigley (Oakland, CA); Jasmine Soria Sears (Kirkland, WA)
Assignee: Facebook Technologies, LLC.
G06T19/006G06T5/009G06T7/74G06T15/205G06T2215/16
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,423,621
App. No.
16/880,207
Granted
Aug 23, 2022
Kind
B1
Abstract

In one embodiment, a computing device may determine a virtual content to be displayed with a scene of a real-world environment. The device may generate an image depicting the virtual content. Using one or more sensors, the device may detect characteristics of the scene of the real-world environment. Based on the image and the characteristics of the scene, the device may determine that a visual enhancement is to be applied to the virtual content depicted in the image to enhance a contrast between the depicted virtual content and the scene. The device may generate a visually-enhanced image depicting the virtual content by applying the visual enhancement to the virtual content depicted in the image. The device may display the visually-enhanced image of the virtual content on a display of the computing device, wherein the scene of the real-world environment is visible through the display.

Claims (61)

1. A method comprising, by a computing device for displaying augmented reality:

determining a virtual content to be displayed with a scene of a real-world environment;

generating an image depicting the virtual content, the image having one or more first characteristics;

detecting, using one or more sensors, one or more second characteristics of the scene of the real-world environment;

comparing the one or more first characteristics of the image with the one or more second characteristics of the scene to determine a perceptibility of the virtual content;

determining, based on the determined perceptibility of the virtual content, that a visual enhancement is to be applied to the virtual content depicted in the image to enhance a contrast between the depicted virtual content and the scene;

generating a visually-enhanced image depicting the virtual content by applying the visual enhancement to the virtual content depicted in the image and reducing a brightness of the scene of the real-world environment; and

displaying the visually-enhanced image of the virtual content on a display of the computing device, wherein the scene of the real-world environment is visible through the display.

2. The method of claim 1 , wherein a contrast between a result of the visually-enhanced image overlaid onto the scene and an underlying region of the scene is perceptibly higher than a contrast between a result of the image overlaid onto the scene and the underlying region.

3. The method of claim 1 , wherein the one or more second characteristics of the scene of the real-world environment comprise one or more of color, luminance, clutter of one or more objects, depth of one or more objects, location of one or more objects, or location of a user.

4. The method of claim 1 , wherein generating the visually-enhanced image depicting the virtual content comprises generating the visually-enhanced image based at least in part on one or more user preferences.

5. The method of claim 1 , wherein the visually-enhanced image includes a line drawn around a perimeter of the virtual content depicted in the visually-enhanced image.

6. The method of claim 1 , further comprising:

identifying an initial area of the scene where the virtual content depicted in the image is to be displayed; and

identifying an alternative area of the scene having a greater contrast with the virtual content than the initial area of the scene;

wherein the virtual content depicted in the visually-enhanced image is displayed over the alternative area of the scene.

7. The method of claim 6 , wherein the alternative area of the scene is within a threshold distance of the initial area of the scene.

8. The method of claim 6 , further comprising:

determining that the alternative area of the scene is free of physical hazards for a user of the computing device.

9. The method of claim 6 , further comprising:

determining that the alternative area of the scene is free of people.

10. The method of claim 1 , wherein applying the visual enhancement comprises adjusting a gamma of the virtual content depicted in the image.

11. The method of claim 1 , wherein applying the visual enhancement comprises:

accessing a normal map of the virtual content; and

generating one or more virtual lighting effects for the virtual content based on the normal map.

12. The method of claim 11 , wherein the virtual content is a first person, and the normal map is generated based on depth data of the first person captured by a first device being utilized by the first person or a second device being utilized by a second person.

13. The method of claim 1 , wherein application of the visual enhancement comprises two or more of:

generating a tone mapping of the virtual content depicted in the image;

generating a line around a perimeter of the virtual content depicted in the image;

increasing a local contrast by applying a high-pass filter to the virtual content depicted in the image; or

increasing a saturation of the virtual content depicted in the image.

14. One or more computer-readable non-transitory storage media embodying software that is operable when executed by a computing device to:

determine a virtual content to be displayed with a scene of a real-world environment;

generate an image depicting the virtual content, the image having one or more first characteristics;

detect, using one or more sensors, one or more second characteristics of the scene of the real-world environment;

compare the one or more first characteristics of the image with the one or more second characteristics of the scene to determine a perceptibility of the virtual content;

determine, based on the determined perceptibility of the virtual content, that a visual enhancement is to be applied to the virtual content depicted in the image to enhance a contrast between the depicted virtual content and the scene;

generate a visually-enhanced image depicting the virtual content by applying the visual enhancement to the virtual content depicted in the image and reducing a brightness of the scene of the real-world environment; and

display the visually-enhanced image of the virtual content on a display of the computing device, wherein the scene of the real-world environment is visible through the display.

15. The media of claim 14 , wherein a contrast between a result of the visually-enhanced image overlaid onto the scene and an underlying region of the scene is perceptibly higher than a contrast between a result of the image overlaid onto the scene and the underlying region.

16. The media of claim 14 , wherein application of the visual enhancement comprises two or more of:

generate a tone mapping of the virtual content depicted in the image;

generate a line around a perimeter of the virtual content depicted in the image;

increase a local contrast by applying a high-pass filter to the virtual content depicted in the image; or

increase a saturation of the virtual content depicted in the image.

17. A system comprising:

one or more non-transitory computer-readable storage media including instructions; and

one or more processors coupled to the storage media, the one or more processors configured to execute the instructions to:

determine a virtual content to be displayed with a scene of a real-world environment;

generate an image depicting the virtual content, the image having one or more first characteristics;

detect, using one or more sensors, one or more second characteristics of the scene of the real-world environment;

compare the one or more first characteristics of the image with the one or more second characteristics of the scene to determine a perceptibility of the virtual content;

determine, based on the determined perceptibility of the virtual content, that a visual enhancement is to be applied to the virtual content depicted in the image to enhance a contrast between the depicted virtual content and the scene;

generate a visually-enhanced image depicting the virtual content by applying the visual enhancement to the virtual content depicted in the image and reducing a brightness of the scene of the real-world environment; and

display the visually-enhanced image of the virtual content on a display of the computing device, wherein the scene of the real-world environment is visible through the display.

18. The system of claim 17 , wherein the one or more processors are further configured to execute the instructions to determine that the visual enhancement is to be applied to the virtual content based at least in part on one or predetermined tags associated with the virtual content.

19. The system of claim 17 , wherein application of the visual enhancement comprises two or more of:

generate a tone mapping of the virtual content depicted in the image;

generate a line around a perimeter of the virtual content depicted in the image;

increase a local contrast by applying a high-pass filter to the virtual content depicted in the image; or

increase a saturation of the virtual content depicted in the image.

Assignments (2)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 22, 2020
From: ATLAS, CHARLENE MARY; BACHY, ROMAIN; MACKENZIE, KEVIN JAMES; MATSUDA, NATHAN; MURDISON, THOMAS SCOTT; QUIGLEY, OCEAN; SEARS, JASMINE SORIA
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 053279/0122 →
Cited By (4)
US 12,512,027 US 12,513,279 US 12,548,268 US 12,695,924