IP Library Granted Patent US 11,196,939
Granted Patent B2
US 11,196,939 · App. 16/876,663 · Granted Dec 7, 2021

Generating light painting images from a sequence of short exposure images

Inventors: Chih-Yao Hsieh (San Jose, CA); Sylvain Paris (Jamaica Plain, MA); Seyed Morteza Safdarnejad (San Jose, CA); Gregg Wilensky (San Francisco, CA)
Assignee: ADOBE INC.
H04N5/2356G06T7/246H04N5/2353H04N5/2354H04N5/2355H04N5/23254
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,196,939
App. No.
16/876,663
Granted
Dec 7, 2021
Kind
B2
Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that generate a virtual long exposure image from a sequence of short exposure images portraying a moving object. In various embodiments, the image transformation system aligns two digital images in the sequence of short exposure images. The image transformation system can determine a motion vector path for the moving object between the first digital image and the second digital image. The image transformation system can also blend pixels along the motion vector path to generate a blended image representative of the motion of the moving object between the first digital image and the second digital image. The image transformation system can generate additional blended images based on consecutive pairs of images in the sequence of digital images and generates a virtual long exposure image by combining the first blended image with the additional blended images.

Claims (45)

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:

determine an alignment among a sequence of digital images;

determine, from the alignment, a motion vector path of moving pixels among the sequence of digital images;

identify, for locations along the motion vector path, respective brightest pixels from portions of digital images within the sequence of digital images corresponding to the motion vector path; and

generate a virtual long exposure image by blending the sequence of digital images together by combining the brightest pixels from the sequence of digital images for the locations along the motion vector path without blending background pixels not associated with the motion vector path.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to convert the digital images in the sequence of digital images into grayscale prior to identifying the brightest pixels from the sequence of digital images for the locations along the motion vector path.

3. The non-transitory computer readable medium of claim 1 , wherein the moving pixels correspond to a light source whose movement is captured by the sequence of digital images.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to extract the sequence of digital images from a video.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to capture the sequence of digital images using a burst mode camera setting.

6. The non-transitory computer readable medium of claim 5 , wherein the sequence of digital images are captured without the use of a tripod.

7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the alignment among the sequence of digital images by:

utilizing a feature-based-alignment model to align pixels between consecutive digital images in the sequence of digital images;

determining a feature-point deficiency; and

applying a pixel-adjusted-gyroscope-alignment model to align the pixels between consecutive digital images in the sequence of digital images in place of the feature-based-alignment model based on determining the feature-point deficiency.

8. The non-transitory computer readable medium of claim 7 , wherein determining the feature-point deficiency comprises determining that a number of feature points within one or more of a set of consecutive digital images in the sequence of digital images does not satisfy a threshold amount of feature points.

9. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the motion vector path by determining, for each pixel in the motion vector path, a horizontal motion and a vertical motion between consecutive images in the sequence of digital images.

10. A system operable to virtual long exposure digital images, the system comprising:

a memory comprising a sequence of digital images;

at least one computing device configured to cause the system to:

determine an alignment among the sequence of digital images;

determine a motion vector path of moving pixels among the sequence of digital images utilizing an optical flow technique and the alignment between the sequence of digital images;

identify, for locations along the motion vector path, respective brightest pixels from portions of digital images within the sequence of digital images corresponding to the motion vector path; and

generate a virtual long exposure image by blending the sequence of digital images together by combining the brightest pixels from the sequence of digital images for the locations along the motion vector path without blending background pixels not associated with the motion vector path.

11. The system of claim 10 , wherein the least one computing device is further configured to cause the system to:

select an image-alignment model from a pixel-adjusted-gyroscope-alignment model and a feature-based-alignment model; and

apply the selected image-alignment model to the sequence of digital images to determine the alignment among the sequence of digital images.

12. The system of claim 10 , wherein the least one computing device is further configured to cause the system to determine the motion vector path by determining, for each of the moving pixels, a horizontal motion and a vertical motion between consecutive digital images.

13. The system of claim 10 , wherein the least one computing device is further configured to cause the system to convert the digital images in the sequence of digital images into grayscale prior to identifying the brightest pixels from the sequence of digital images for the locations along the motion vector path.

14. The system of claim 10 , wherein the least one computing device is further configured to cause the system to:

generate a first blended digital image by combining a first pair of consecutive digital images in the sequence of digital images by combining a brightest pixel for each location in the motion vector path from the first pair of consecutive digital images;

generate a second blended digital image by combining a second pair of consecutive digital images in the sequence of digital images by combining a brightest pixel for each location in the motion vector path from the second pair of consecutive digital images; and

generate the virtual long exposure image by blending the sequence of digital images together by combining the first blended digital image and the second blended digital image by combining a brightest pixel for each location in the motion vector path from the first blended digital image and the second blended digital image.

15. The system of claim 10 , wherein the least one computing device is further configured to cause the system to identify the brightest pixels from the sequence of digital images for the locations along the motion vector path by comparing a brightness of a pixel at a first location in the motion vector path from each digital image in the sequence of digital images.

16. A method of generating virtual long exposure images comprising:

determining an alignment among a sequence of digital images;

determining, from the alignment a motion vector path of moving pixels among the sequence of digital images;

identifying, for locations along the motion vector path, respective brightest pixels from portions of digital images within the sequence of digital images corresponding to the motion vector path; and

generating a virtual long exposure image by blending the sequence of digital images together by combining the brightest pixels from the sequence of digital images for the locations along the motion vector path without blending background pixels not associated with the motion vector path.

17. The method of claim 16 , further comprising converting the digital images in the sequence of digital images into grayscale prior to identifying the brightest pixels from the sequence of digital images for the locations along the motion vector path.

18. The method of claim 16 , wherein generating the virtual long exposure image comprises generating a virtual light-painting long exposure image.

19. The method of claim 16 , wherein determining the alignment between the sequence of digital images comprises:

utilizing a feature-based-alignment model to align pixels between consecutive digital images in the sequence of digital images;

determining a feature-point deficiency; and

applying a pixel-adjusted-gyroscope-alignment model to align pixels between consecutive digital images in the sequence of digital images in place of the feature-based-alignment model based on determining the feature-point deficiency.

20. The method of claim 16 , wherein determining the motion vector path comprises determining, for each pixel in the motion vector path, a horizontal motion and a vertical motion between consecutive images in the sequence of digital images utilizing an optical flow technique.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 18, 2020
From: HSIEH, CHIH-YAO; PARIS, SYLVAIN; SAFDARNEJAD, SEYED MORTEZA; WILENSKY, GREGG
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 052688/0494 →
CHANGE OF NAME Recorded May 18, 2020
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 052691/0806 →
Continuity (2)
Division 16149904 · Oct 2, 2018
Related Publication 20200280670A1 · Sep 3, 2020
Cited By (1)
US 12,739,525