Granted Patent B2

US 12,639,868 · App. 18/726,060 · Granted May 26, 2026

Inventor: Dongeek Shin (Mountain View, CA)

Assignee: Google LLC

G06T11/60G06T7/40G06V10/764G06V10/82G06V20/70G06F3/14G06T2210/52

Abstract

Claims (82)

1 . A method comprising:

capturing, by an image sensor of a computing device, an image frame;

determining, by one or more processors of the computing device and based on the image frame, one or more characteristics of light produced by one or more light sources;

identifying, by the one or more processors, one or more items in the image frame;

determining, by the one or more processors, one or more labels that correspond to the one or more items identified in the image frame;

generating, based on the one or more labels, a mask image corresponding to the one or more light sources;

generating, by the one or more processors, an adjusted image by at least applying a light source image to an image of an object to simulate effects of the one or more light sources on the object, wherein the light source image is generated based on the one or more characteristics of light produced by the one or more light sources associated with the mask image; and

outputting, by the one or more processors and for display, the adjusted image.

2 . The method of claim 1 , wherein the identifying the one or more items further comprises:

identifying, by the one or more processors, the one or more items within the image frame with a class object detector.

3 . The method of claim 1 , wherein generating the mask image further comprises:

generating, by the one or more processors with image texture analysis, a texture map of boundaries corresponding to surface textures from the image; and

generating, by the one or more processors, the mask image associated with the one or more light sources based on the texture map.

4 . The method of claim 1 , wherein generating the mask image further comprises:

sending the one or more characteristics of light produced by one or more light sources to a remote computing device to identify one or more items in the image frame corresponding to the one or more light sources, and to generate the mask image; and

receiving the mask image from the remote computing device.

5 . The method of claim 1 , further comprising:

computing the light source image with a machine learning model trained to generate the light source image based on the mask image.

6 . The method of claim 1 , further comprising:

sending, from the computing device to a remote computing device, the mask image; and

receiving, by the computing device, and from the remote computing device, the light source image generated using the mask image.

7 . The method of claim 1 , further comprising:

determining, by the computing device, whether movement of a display device satisfies a threshold value;

responsive to determining that the movement is greater than the threshold value, capturing, by the computing device, image frames at a rate of at least 30 cycles per second (Hz); and

responsive to determining that the movement is less than the threshold value, capturing, by the computing device, image frames at a rate less than 30 cycles per second (Hz).

8 . The method of claim 7 , further comprising:

receiving, by the computing device, motion data generated by a motion sensor of the display device,

wherein determining whether the movement of the display satisfies the threshold value is based on the motion data.

9 . A computing device comprising:

an image sensor configured to capture an image frame;

a display device;

a memory; and

one or more processors operably coupled to the memory and configured to:

determine, based on the image frame, one or more characteristics of light produced by one or more light sources;

identify one or more items in the image frame;

determine one or more labels that correspond to the one or more items identified in the image frame;

generate, based on the one or more labels, a mask image corresponding to the one or more light sources;

apply a light source image, generated with the one or more characteristics of light associated with the one or more light sources associated with the mask image, to an image of an object to generate an adjusted image to simulate effects of the one or more light sources on the object; and

output, for display by the display device, the adjusted image.

10 . The computing device of claim 9 , wherein the one or more processors are configured to generate the mask image by at least being configured to:

send, to a remote computing device, the one or more characteristics of light produced by one or more light sources remote computing device to identify one or more items in the image frame corresponding to the one or more light sources, and to generate the mask image; and

receive, from the remote computing device, the mask image generated by at least applying a machine learning model to the mask image.

11 . The computing device of claim 9 , wherein the one or more processors are configured to generate the mask image by at least being configured to:

generate a texture map of boundaries corresponding to surface textures from the image; and

generate the mask image associated with the one or more light sources based on the texture map.

12 . The computing device of claim 9 , wherein the one or more processors are configured to generate the mask image by at least being configured to:

identify one or more items in the image frame;

determine one or more labels that correspond to the one or more items identified in the image frame; and

generate, based on the one or more labels, the mask image corresponding to the one or more light sources.

13 . The computing device of claim 9 , further comprising:

one or more motion sensors that generate motion data based on movement of the computing device,

wherein the one or more processors are further configured to:

determine, based on the motion data, whether movement of the display device satisfies a threshold value;

responsive to determining that the movement is greater than the threshold value, capture, using the image sensor, image frames at a rate of at least 30 cycles per second (Hz); and

responsive to determining that the movement is less than the threshold value, capture, using the image sensor, image frames at a rate less than 30 cycles per second (Hz).

14 . A non-transitory computer-readable medium encoded with instructions that, when executed by one or more processors of a computing device, cause the one or more processors to:

capture, using an image sensor of the computing device, an image frame;

determine, based on the image frame, one or more characteristics of light produced by one or more light sources;

identify one or more items in the image frame;

determine one or more labels that correspond to the one or more items identified in the image frame;

generate, based on the one or more labels, a mask image corresponding to the one or more light sources;

generate an adjusted image by at least applying a light source image to an image of an object to simulate effects of the one or more light sources on the object, wherein the light source image is generated based on the one or more characteristics of light produced by the one or more light sources associated with the mask image; and

output, for display, the adjusted image.

15 . The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the one or more processors to:

determine whether movement of a display device satisfies a threshold value;

responsive to determining that the movement is greater than the threshold value, capture image frames at a rate of at least 30 cycles per second (Hz); and

responsive to determining that the movement is less than the threshold value, capture image frames at a rate less than 30 cycles per second (Hz).

16 . The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the one or more processors to generate the mask image by at least causing the one or more processors to:

send, to a remote computing device, the one or more characteristics of light produced by one or more light sources remote computing device to identify one or more items in the image frame corresponding to the one or more light sources, and to generate the mask image; and

receive, from the remote computing device, the mask image generated by at least applying a machine learning model to the mask image.

17 . The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the one or more processors to generate the mask image by at least causing the one or more processors to:

generate a texture map of boundaries corresponding to surface textures from the image; and

generate the mask image associated with the one or more light sources based on the texture map.

18 . The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the one or more processors to generate the mask image by at least causing the one or more processors to:

identify one or more items in the image frame;

determine one or more labels that correspond to the one or more items identified in the image frame; and

generate, based on the one or more labels, the mask image corresponding to the one or more light sources.

19 . The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the one or more processors to:

generate, using one or more motion sensors, motion data based on movement of the computing device,

determine, based on the motion data, whether movement of a display device of the computing device satisfies a threshold value;

responsive to determining that the movement is greater than the threshold value, capture, using the image sensor, image frames at a rate of at least 30 cycles per second (Hz); and

responsive to determining that the movement is less than the threshold value, capture, using the image sensor, image frames at a rate less than 30 cycles per second (Hz).

Continuity (1)

References Cited (41)