IP Library Granted Patent US 10,523,916
Granted Patent B2
US 10,523,916 · App. 15/975,727 · Granted Dec 31, 2019

Modifying images with simulated light sources

Inventors: Nils Kokemohr (Hamburg, DE); Daniel Fenner (Hamburg, DE)
Assignee: Google LLC
H04N13/207
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,523,916
App. No.
15/975,727
Granted
Dec 31, 2019
Kind
B2
Abstract

Implementations relate to modifying images with simulated light sources. In some implementations, a method includes receiving input indicating a location on a 2-D displayed image at which to position a light source element, and adjusting pixel values of the image to cause an illumination effect in the image based on the position of the light source element and based on a direction of illumination of the simulated light source with respect to one or more surfaces depicted in the image.

Claims (34)

1. A computer-implemented method comprising:

receiving input indicating a location on a displayed two-dimensional (2-D) image at which to position a light source element that provides a simulated light source; and

adjusting pixel values of the image to cause an illumination effect in the image based on the position of the light source element and based on an angle of illumination of the simulated light source with respect to one or more surfaces depicted in the image,

wherein adjusting pixel values includes determining an estimated three-dimensional (3-D) model of a scene depicted in the image, wherein determining the estimated 3-D model includes estimating depths into the scene of one or more non-face objects depicted in the image based on locations of one or more faces depicted in the image compared to locations of the one or more non-face objects depicted in the image, wherein particular non-face objects of the one or more non-face objects that are adjacent to the one or more faces are designated to have a greater depth into the scene than the one or more faces adjacent to the particular non-face objects.

2. The method of claim 1 wherein adjusting the pixel values further includes determining one or more surface angles of the one or more surfaces depicted in the image by estimating local normal vectors for the one or more surfaces.

3. The method of claim 2 wherein estimating the local normal vectors for the one or more surfaces includes using one or more of the depths into the scene of the one or more non-face objects.

4. The method of claim 2 wherein determining the one or more surface angles is based on a predefined map of face surfaces adapted to the one or more faces depicted in the image.

5. The method of claim 2 further comprising adapting a size and a shape of a predefined map of face surfaces to the one or more faces depicted in the image, wherein the estimating the local normal vectors for the one or more surfaces includes estimating the local normal vectors for surfaces of the one or more faces based on the predefined map adapted to the one or more faces depicted in the image.

6. The method of claim 5 wherein the predefined map is a standard map of face surfaces.

7. The method of claim 1 wherein adjusting pixel values further includes determining image depth from a camera that captured the image to one or more objects depicted in the image based on a brightness of one or more object surfaces of the one or more objects.

8. The method of claim 7 wherein the brightness of the one or more object surfaces is caused by a flash in effect when the image was captured.

9. A computer-implemented method comprising:

receiving input indicating a location on a displayed two-dimensional (2-D) image at which to position a light source element that provides a simulated light source;

adjusting pixel values of the image to cause an illumination effect in the image based on the position of the light source element and based on an angle of illumination of the simulated light source with respect to one or more surfaces depicted in the image, wherein adjusting the pixel values includes determining one or more surface angles of the one or more surfaces depicted in the image; and

adapting a size and a spacing of a predefined map of face surfaces to one or more faces depicted in the image, wherein determining the one or more surface angles is based on the predefined map of face surfaces adapted to the one or more faces depicted in the image.

10. The method of claim 9 wherein adapting the size and the spacing of the predefined map of face surfaces to the one or more faces depicted in the image includes adapting the spacing of the predefined map of face surfaces to facial landmarks of the one or more faces.

11. The method of claim 9 wherein the one or more surface angles are determined by estimating local normal vectors for the one or more surfaces.

12. The method of claim 11 wherein estimating the local normal vectors for the one or more surfaces includes estimating the local normal vectors for one or more surfaces of human faces based on the predefined map of face surfaces adapted to the one or more faces depicted in the image.

13. The method of claim 12 wherein the predefined map is a standard map of face surfaces.

14. The method of claim 11 wherein adjusting pixel values includes determining an estimated three-dimensional (3-D) model of a scene depicted in the image, wherein determining the estimated 3-D model includes estimating depths into the scene of one or more non-face objects depicted in the image based on locations of the one or more faces depicted in the image compared to locations of the one or more non-face objects depicted in the image, wherein particular non-face objects of the one or more non-face objects that are adjacent to the one or more faces are designated to have a greater depth into the scene than the one or more faces adjacent to the particular non-face objects,

wherein estimating the local normal vectors for the one or more surfaces includes using one or more of the depths into the scene of the one or more non-face objects.

15. The method of claim 9 wherein adjusting the pixel values of the image further includes estimating the one or more surfaces depicted in the image based on analysis of image pixels of the image without reference to data describing three-dimensional (3-D) features of a scene depicted by the image.

16. A system comprising:

a storage device storing a two-dimensional (2-D) image; and

at least one processor accessing the storage device and configured to perform operations comprising:

causing the image to be displayed on a display device;

receiving input indicating a location on a displayed image at which to position a light source element that provides a simulated light source;

adjusting pixel values of the image to cause an illumination effect in the image based on the position of the light source element and based on an angle of illumination of the simulated light source with respect to one or more surfaces depicted in the image, wherein adjusting the pixel values includes determining one or more surface angles of the one or more surfaces depicted in the image; and

adapting a size and a spacing of a predefined map of face surfaces to one or more faces depicted in the image, wherein determining the one or more surface angles is based on the predefined map of face surfaces adapted to the one or more faces depicted in the image.

17. The system of claim 16 wherein the operation of adapting the size and the spacing of the predefined map of face surfaces to the one or more faces depicted in the image includes adapting the spacing of the predefined map of face surfaces to facial landmarks of the one or more faces.

18. The system of claim 16 the one or more surface angles are determined by the at least one processor performing operations including estimating local normal vectors for the one or more surfaces.

19. The system of claim 18 wherein the operation of estimating the local normal vectors for the one or more surfaces includes estimating the local normal vectors for one or more surfaces of human faces based on the predefined map of face surfaces adapted to the one or more faces depicted in the image.

20. The system of claim 18 wherein the operation of adjusting pixel values includes determining an estimated three-dimensional (3-D) model of a scene depicted in the image, wherein determining the estimated 3-D model includes estimating depths into the scene of one or more non-face objects depicted in the image based on locations of the one or more faces depicted in the image compared to locations of the one or more non-face objects depicted in the image, wherein particular non-face objects of the one or more non-face objects that are adjacent to the one or more faces are designated to have a greater depth into the scene than the one or more faces adjacent to the particular non-face objects,

wherein estimating the local normal vectors for the one or more surfaces includes using one or more of the depths into the scene of the one or more non-face objects.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 9, 2018
From: KOKEMOHR, NILS; FENNER, DANIEL
To: GOOGLE INC.
Reel/Frame 045758/0996 →
CHANGE OF NAME Recorded May 9, 2018
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 046125/0119 →
Continuity (3)
Continuation 14469540 · Aug 26, 2014
Provisional Application 62018367 · Jun 27, 2014
Related Publication 20180262671A1 · Sep 13, 2018