IP Library Granted Patent US 11,436,814
Granted Patent B2
US 11,436,814 · App. 17/328,998 · Granted Sep 6, 2022

Adjusted overlaid images in scenes

Inventors: Barry David Silverstein (Kirkland, WA); Romain Bachy (Seattle, WA); Edward Buckley (Redmond, WA)
Assignee: Meta Platforms Technologies, LLC
G06T19/20G02B27/017G06T7/70G06T19/006G02B2027/014G02B2027/0112G06T2207/10024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,436,814
App. No.
17/328,998
Granted
Sep 6, 2022
Kind
B2
Abstract

Adjusted overlaid images are generated in response to at least one color reference object or color reference point. A display is driven to present the adjusted overlaid image.

Claims (35)

1. A device comprising:

a display configured to generate overlaid images for overlaying with real-world scenes;

a light sensor configured to capture a light measurement of scene light from an external environment of the device, wherein the light sensor includes an image sensor oriented to capture a scene image as the light measurement; and

processing logic configured to:

determine a color space adjustment value in response to the light measurement;

generate adjusted overlaid images by adjusting an image color of at least a portion of the overlaid images in response to the color space adjustment value; and

drive the display to present the adjusted overlaid images overlaid with the real-world scenes,

wherein the determining the color space adjustment value in response to the light measurement includes:

identifying a color reference point in the scene image; and

determining a difference vector between pixel values of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein the image color of the at least a portion of the overlaid image is adjusted based on the difference vector.

2. The device of claim 1 , wherein identifying the color reference point includes identifying a whitest object in the scene image, and wherein the calibrated color value is a white color reference point.

3. The device of claim 1 , wherein the processing logic is configured to receive the light measurement from the light sensor.

4. The device of claim 1 , wherein the device is a head mounted display (HMD).

5. A computer-implemented method comprising:

capturing a light measurement with a light sensor of a head mounted display (HMD) configured to receive scene light from an external environment of the HMD;

determining a color space adjustment value in response to the light measurement;

generating an adjusted overlaid image by adjusting an image color of at least a portion of an overlaid image in response to the color space adjustment value; and

driving a display of the HMD to present the adjusted overlaid image overlaid with real-world scenes viewable to a user of the HMD,

wherein the light sensor includes an image sensor and the image sensor is oriented to capture a scene image that includes a color reference object included in the HMD, the light measurement including the scene image,

and wherein determining the color space adjustment value in response to the light measurement includes:

identifying a color reference point in the scene image; and

determining a difference vector between pixel values of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein the image color of the at least a portion of the overlaid image is adjusted based on the difference vector.

6. The computer-implemented method of claim 5 , wherein adjusting the image color includes adjusting sub-pixel values of the overlaid image based on the color space adjustment value.

7. The computer-implemented method of claim 5 , wherein the display includes a first, second, and third light source, and wherein adjusting the image color includes adjusting an intensity of at least one of the first, second, or third light source based on the color space adjustment value.

8. A computer-implemented method comprising:

receiving a scene image of a real-world scene;

identifying an overlay zone of the real-world scene that an overlaid image will occupy;

identifying at least one color reference point in the scene image of the real-world scene;

selecting a closest color reference point from the at least one color reference point that is closest to the overlay zone;

generating a color space adjustment map for the overlaid image utilizing the closest color reference point;

applying the color space adjustment map to the overlaid image to generate an adjusted overlaid image by adjusting an image color of at least a portion of the overlaid image; and

driving a display to present the adjusted overlaid image to be overlaid with real-world scenes viewable by a user.

9. The computer-implemented method of claim 8 , wherein identifying the at least one color reference point includes matching objects in the real-world scene to a known color reference object included in an array of prior photos that are accessible to a user.

10. The computer-implemented method of claim 8 , wherein the color reference point includes a sclera.

11. The computer-implemented method of claim 8 , wherein the scene image is captured by a front-facing camera of a head mounted display.

Assignments (2)
CHANGE OF NAME Recorded Jun 1, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060246/0845 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 21, 2021
From: SILVERSTEIN, BARRY DAVID; BACHY, ROMAIN; BUCKLEY, EDWARD
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 056601/0592 →
Continuity (3)
Continuation 16710153 · Dec 11, 2019
Provisional Application 62867525 · Jun 27, 2019
Related Publication 20210279972A1 · Sep 9, 2021