IP Library Granted Patent US 11,544,831
Granted Patent B2
US 11,544,831 · App. 16/984,992 · Granted Jan 3, 2023

Utilizing an image exposure transformation neural network to generate a long-exposure image from a single short-exposure image

Inventors: Yilin Wang (San Jose, CA); Zhe Lin (Fremont, CA); Zhaowen Wang (San Jose, CA); Xin Lu (Mountain View, CA); Xiaohui Shen (San Jose, CA); Chih-Yao Hsieh (San Jose, CA)
Assignee: Adobe Inc.
G06T5/50G06T2200/21G06T2207/10004G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,544,831
App. No.
16/984,992
Granted
Jan 3, 2023
Kind
B2
Abstract

The present disclosure relates to training and utilizing an image exposure transformation network to generate a long-exposure image from a single short-exposure image (e.g., still image). In various embodiments, the image exposure transformation network is trained using adversarial learning, long-exposure ground truth images, and a multi-term loss function. In some embodiments, the image exposure transformation network includes an optical flow prediction network and/or an appearance guided attention network. Trained embodiments of the image exposure transformation network generate realistic long-exposure images from single short-exposure images without additional information.

Claims (36)

1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer system to:

receive a short-exposure target image;

determine an attention map for the short-exposure target image utilizing an appearance guided attention neural network, the attention map comprising transformation regions and corresponding magnitudes for the short-exposure target image;

generate a synthesized long-exposure image from the short-exposure target image and the attention map utilizing an image exposure generator neural network by applying long-exposure effects to the short-exposure target image based on the transformation regions and the corresponding magnitudes; and

display, to a client device, the synthesized long-exposure image within a graphical user interface.

2. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to receive the short-exposure target image from a live camera capture of a mobile client device.

3. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to receive the short-exposure target image from a storage medium.

4. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to determine the attention map for the short-exposure target image by generating spatial context predictions of the short-exposure target image utilizing an appearance guided attention neural network.

5. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to determine magnitudes corresponding to the transformation regions within the attention map that comprise weighted parameters indicating motions influences within the transformation regions.

6. The non-transitory computer-readable medium of claim 5 , further comprising added instructions that, when executed by the at least one processor, cause the computer system to generate the synthesized long-exposure image utilizing the image exposure generator neural network by applying the long-exposure effects to the transformation regions of the short-exposure target image according to the weighted parameters of the corresponding magnitudes.

7. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to generate the synthesized long-exposure image utilizing the image exposure generator neural network further based on input modifying an exposure magnitude to one or more transformation regions of the attention map.

8. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to generate the synthesized long-exposure image utilizing the image exposure generator neural network by applying the long-exposure effects to learned features of the short-exposure target image based on attention weights within the attention map in accordance with the transformation regions and the corresponding magnitudes.

9. The non-transitory computer-readable medium of claim 1 , further comprising additional instructions that, when executed by the at least one processor, cause the computer system to generate the synthesized long-exposure image utilizing the image exposure generator neural network further based on an optical flow prediction.

10. The non-transitory computer-readable medium of claim 9 , further comprising added instructions that, when executed by the at least one processor, cause the computer system to generate the optical flow prediction utilizing an optical flow prediction network that indicates pixel movement and direction data in the short-exposure target image.

11. A system for synthesizing long-exposure images from single short-exposure images comprising:

a memory device comprising:

a short-exposure target image;

an appearance guided attention neural network; and

an image exposure generator neural network;

at least one computing device configured to cause the system to:

generate an attention map that comprises a spatial context of the short-exposure target image utilizing the appearance guided attention neural network on the short-exposure target image;

generate a synthesized long-exposure image from the short-exposure target image and the attention map, utilizing the image exposure generator neural network, by applying long-exposure effects to learned features of the short-exposure target image based on attention weights within the attention map; and

provide the synthesized long-exposure image for display.

12. The system of claim 11 , wherein the at least one computing device is further configured to cause the system to determine the attention map for the short-exposure target image by generating spatial context predictions of the short-exposure target image utilizing an appearance guided attention neural network.

13. The system of claim 11 , wherein the at least one computing device is further configured to cause the system to determine the attention weights within the attention map to indicate magnitudes for corresponding transformation regions within the attention map.

14. The system of claim 13 , wherein the at least one computing device is further configured to cause the system to generate the synthesized long-exposure image utilizing the image exposure generator neural network by applying the long-exposure effects to the transformation regions of the short-exposure target image according to corresponding magnitudes as indicated by the attention weights.

15. The system of claim 11 , wherein the at least one computing device is further configured to cause the system to provide the synthesized long-exposure image for display on a graphical user interface of a mobile client device.

16. In a digital medium environment for editing electronic images, a computer-implemented method of transforming exposure time of images, comprising:

receiving a short-exposure target image;

determining an attention map for the short-exposure target image utilizing an appearance guided attention neural network that indicates transformation regions and corresponding magnitudes in the short-exposure target image;

generating a synthesized long-exposure image from the short-exposure target image, utilizing an image exposure generator neural network, by applying long-exposure effects to the transformation regions of the short-exposure target image in accordance with the corresponding magnitudes as indicated in the attention map; and

providing the synthesized long-exposure image for display within a graphical user interface.

17. The method of claim 16 , further comprising receiving the short-exposure target image from a live camera capture of a mobile client device.

18. The method of claim 16 , further comprising generating the synthesized long-exposure image utilizing the image exposure generator neural network further based on input modifying an exposure magnitude to one or more transformation regions of the attention map.

19. The method of claim 16 , further comprising generating the synthesized long-exposure image utilizing the image exposure generator neural network further based on an optical flow prediction.

20. The method of claim 16 , further comprising determining, for each transformation region of the transformation regions, one or more features, wherein the corresponding magnitudes comprise an long-exposure importance of the one or more features within a transformation region.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 4, 2020
From: WANG, YILIN; LIN, ZHE; WANG, ZHAOWEN; LU, XIN; SHEN, XIAOHUI; HSIEH, CHIH-YAO
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 053399/0883 →
CHANGE OF NAME Recorded Aug 4, 2020
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 053399/0982 →
Continuity (2)
Continuation 15962735 · Apr 25, 2018
Related Publication 20200372622A1 · Nov 26, 2020