IP Library › Granted Patent US 11,776,200
Granted Patent B2
US 11,776,200 · App. 17/523,210 · Granted Oct 3, 2023

Image relighting

Inventors: Xianling Zhang (San Jose, CA); Nathan Tseng (Canton, MI); Nikita Jaipuria (Union City, CA); Rohan Bhasin (Santa Clara, CA)
Assignee: Ford Global Technologies, LLC
G06T15/50G06T15/005G06T15/205G06V10/758
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,776,200
App. No.
17/523,210
Granted
Oct 3, 2023
Kind
B2
Abstract

A computer includes a processor and a memory storing instructions executable by the processor to receive a plurality of first images of an environment in a first lighting condition, classify pixels of the first images into categories, mask the pixels belonging to at least one of the categories from the first images, generate a three-dimensional representation of the environment based on the masked first images, and generate a second image of the environment in a second lighting condition based on the three-dimensional representation and on a first one of the first images.

Claims (30)

1. A computer comprising a processor and a memory storing instructions executable by the processor to:

receive a plurality of first images of an environment in a first lighting condition;

classify pixels of the first images into categories;

mask the pixels belonging to at least one of the categories from the first images;

generate a three-dimensional representation of the environment based on the masked first images;

generate a first shadow mask of the first one of the first images, wherein generating the first shadow mask includes executing a second machine-learning algorithm, and the first images are inputs to the second machine-learning algorithm;

generate a second shadow mask of the environment in the second lighting condition from a perspective of the first one of the first images, wherein generating the second shadow mask includes executing a third machine-learning algorithm, and the first images are inputs to the third machine-learning algorithm; and

generate a second image of the environment in a second lighting condition based on the three-dimensional representation, on a first one of the first images, on the first shadow mask, and on the second shadow mask, wherein generating the second image includes executing a first machine-learning algorithm; and the first one of the first images, the first shadow mask, and the second shadow mask are inputs to the first machine-learning algorithm.

2. The computer of claim 1 , wherein the second image and the first one of the first images have a same perspective of the environment.

3. The computer of claim 1 , wherein the instructions further include instructions to generate a plurality of second images including the second image based on the three-dimensional representation and on the first images, the second images being in the second lighting condition.

4. The computer of claim 3 , wherein each second image has a same perspective of the environment as respective ones of the first images.

5. The computer of claim 1 , wherein the at least one of the categories includes sky.

6. The computer of claim 1 , wherein the first images are of the environment at a series of points along a path through the environment.

7. The computer of claim 6 , wherein the path extends along a roadway of the environment.

8. The computer of claim 1 , wherein the three-dimensional representation is a mesh.

9. The computer of claim 8 , wherein generating the mesh includes generating a point cloud based on the masked first images and generating the mesh based on the point cloud.

10. The computer of claim 9 , wherein generating the point cloud includes executing a fourth machine-learning algorithm, and the masked first images are inputs to the fourth machine-learning algorithm.

11. The computer of claim 1 , wherein the instructions further include instructions to generate a shadow mask of the environment in the second lighting condition from a perspective of the first one of the first images, and generating the second image is based on the shadow mask.

12. The computer of claim 1 , wherein the second lighting condition includes a light direction, and generating the second shadow mask includes determining shadow locations by projecting objects in the three-dimensional representation along the light direction.

13. The computer of claim 1 , wherein the second lighting condition includes a light direction, generating the second shadow mask includes determining a preliminary second shadow mask having shadow locations by projecting objects in the three-dimensional representation along the light direction, and the preliminary second shadow mask is an input to the second machine-learning algorithm.

14. The computer of claim 1 , wherein the instructions further include instructions to generate a reflectance map of the environment from a perspective of the first one of the first images based on the three-dimensional representation, the reflectance map is a map of specular reflection direction based on a light direction of the second lighting condition, and generating the second image is based on the reflectance map.

15. The computer of claim 14 , wherein the instructions further include instructions to generate a normal map of the environment from the perspective of the first one of the first images based on the three-dimensional representation, and generating the reflectance map is based on the normal map and the second lighting condition.

16. A method comprising:

receiving a plurality of first images of an environment in a first lighting condition;

classifying pixels of the first images into categories;

masking the pixels belonging to at least one of the categories from the first images;

generating a three-dimensional representation of the environment based on the masked first images;

generating a first shadow mask of the first one of the first images, wherein generating the first shadow mask includes executing a second machine-learning algorithm, and the first images are inputs to the second machine-learning algorithm;

generating a second shadow mask of the environment in the second lighting condition from a perspective of the first one of the first images, wherein generating the second shadow mask includes executing a third machine-learning algorithm, and the first images are inputs to the third machine-learning algorithm; and

generating a second image of the environment in a second lighting condition based on the three-dimensional representation, on a first one of the first images, on the first shadow mask, and on the second shadow mask, wherein generating the second image includes executing a first machine-learning algorithm; and the first one of the first images, the first shadow mask, and the second shadow mask are inputs to the first machine-learning algorithm.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 10, 2021
From: ZHANG, XIANLING; TSENG, NATHAN; JAIPURIA, NIKITA; BHASIN, ROHAN
To: FORD GLOBAL TECHNOLOGIES, LLC
Reel/Frame 058072/0148 →
Continuity (1)
Related Publication 20230143816A1 · May 11, 2023