Low-light and/or long exposure image capture mode
An incoming image stream may be obtained from an image capture device operating in low-light conditions and/or a simulated long exposure image capture mode. As images are obtained, a weighting operation may be performed on the pixels of the captured images to generate and/or update an accumulative weight map, wherein the weighting is based, e.g., on the proximity of the captured pixels' values to the respective image capture device's maximum observable pixel value. As batches of images are obtained, they may be fused, e.g., according to the accumulative weight map, in a memory-efficient manner that places an upper limit on the overall memory footprint of the fusion operations, to simulate an actual long exposure image capture. In some embodiments, the weight map may be stored at a lower resolution than the obtained images and then upscaled, e.g., via the use of guided filters, before being applied in the fusion operations.
1. A device, comprising:
a memory;
one or more image capture devices;
a display;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a first batch of images captured by the one or more image capture devices;
perform a pixel weighting operation on each image in the first batch of images to output a weight map;
perform a first fusion operation on the first batch of images according to the weight map to output a first fused image;
designate the first fused image as a current intermediate fusion image;
successively perform a subsequent fusion sequence at least one time, wherein each subsequent fusion sequence comprises:
obtaining a subsequent batch of images captured by the one or more image capture devices;
performing a pixel weighting operation on each image in the subsequent batch of images to output an updated weight map;
performing a subsequent fusion operation on: the subsequent batch of images and the current intermediate fusion image according to the updated weight map to output an updated fused image; and
designating the updated fused image as the current intermediate fusion image; and
following completion of the last subsequent fusion sequence, store the current intermediate fusion image as an output fused image in the memory.
2. The device of claim 1 , wherein the instructions further comprise instructions causing the one or more processors to:
detect that the one or more image capture devices have been stationary for a threshold amount of time prior to obtaining the first batch of images.
3. The device of claim 1 , wherein the weight map comprises a plurality of fusion weight values, and wherein the weight map has a lower resolution than the obtained images.
4. The device of claim 3 , wherein the instructions to perform a subsequent fusion operation according to the updated weight map further comprise instructions causing the one or more processors to:
upscale the updated weight map to the resolution of the obtained images prior to performing the subsequent fusion operation.
5. The device of claim 4 , wherein the instructions to upscale the updated weight map further comprise instructions to:
upscale the updated weight map using one or more filters.
6. The device of claim 5 , wherein the one or more filters are guided, at least in part, by the current intermediate fusion image output by the previous subsequent fusion sequence.
7. The device of claim 1 , wherein the pixel weighting operation is configured to assign a weight to a pixel based, at least in part, on how close the pixel's value is to a maximum observable pixel value for the image capture device that captured the pixel.
8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a first batch of images captured by one or more image capture devices;
perform a pixel weighting operation on each image in the first batch of images to output a weight map;
perform a first fusion operation on the first batch of images according to the weight map to output a first fused image;
designate the first fused image as a current intermediate fusion image;
successively perform a subsequent fusion sequence at least one time, wherein each subsequent fusion sequence comprises:
obtaining a subsequent batch of images captured by the one or more image capture devices;
performing a pixel weighting operation on each image in the subsequent batch of images to output an updated weight map;
performing a subsequent fusion operation on: the subsequent batch of images and the current intermediate fusion image according to the updated weight map to output an updated fused image; and
designating the updated fused image as the current intermediate fusion image; and
following completion of the last subsequent fusion sequence, store the current intermediate fusion image as an output fused image in a memory.
9. The non-transitory program storage device of claim 8 , wherein the weight map comprises a plurality of fusion weight values, and wherein the weight map has a lower resolution than the obtained images.
10. The non-transitory program storage device of claim 9 , wherein the instructions to perform a subsequent fusion operation according to the updated weight map further comprise instructions causing the one or more processors to:
upscale the updated weight map to the resolution of the obtained images prior to performing the subsequent fusion operation.
11. The non-transitory program storage device of claim 10 , wherein the instructions to upscale the updated weight map further comprise instructions to:
upscale the updated weight map using one or more filters.
12. The non-transitory program storage device of claim 11 , wherein the one or more filters are guided, at least in part, by the current intermediate fusion image output by the previous subsequent fusion sequence.
13. The non-transitory program storage device of claim 8 , wherein the pixel weighting operation is configured to assign a weight to a pixel based, at least in part, on how close the pixel's value is to a maximum observable pixel value for the image capture device that captured the pixel.
14. A method, comprising:
obtaining a first batch of images captured by one or more image capture devices;
performing a pixel weighting operation on each image in the first batch of images to output a weight map;
performing a first fusion operation on the first batch of images according to the weight map to output a first fused image;
designating the first fused image as a current intermediate fusion image;
successively performing a subsequent fusion sequence at least one time, wherein each subsequent fusion sequence comprises:
obtaining a subsequent batch of images captured by the one or more image capture devices;
performing a pixel weighting operation on each image in the subsequent batch of images to output an updated weight map;
performing a subsequent fusion operation on: the subsequent batch of images and the current intermediate fusion image according to the updated weight map to output an updated fused image; and
designating the updated fused image as the current intermediate fusion image; and
following completion of the last subsequent fusion sequence, storing the current intermediate fusion image as an output fused image in a memory.
15. The method of claim 14 , further comprising:
detecting that the one or more image capture devices have been stationary for a threshold amount of time prior to obtaining the first batch of images.
16. The method of claim 14 , wherein the weight map comprises a plurality of fusion weight values, and wherein the weight map has a lower resolution than the obtained images.
17. The method of claim 16 , wherein performing a subsequent fusion operation according to the updated weight map further comprises:
upscaling the updated weight map to the resolution of the obtained images prior to performing the subsequent fusion operation.
18. The method of claim 17 , wherein upscaling the updated weight map further comprises:
upscaling the updated weight map using one or more filters.
19. The method of claim 18 , wherein the one or more filters are guided, at least in part, by the current intermediate fusion image output by the previous subsequent fusion sequence.
20. The method of claim 14 , wherein the pixel weighting operation is configured to assign a weight to a pixel based, at least in part, on how close the pixel's value is to a maximum observable pixel value for the image capture device that captured the pixel.