Apparatus and method for detail enhancement in super-resolution imaging using mobile electronic device

1. A method comprising:

obtaining multiple image frames of a scene using at least one camera of an electronic device; and

processing the multiple image frames of the scene to generate a higher-resolution image of the scene, wherein processing the multiple image frames of the scene comprises:

generating an initial estimate of the scene based on the multiple image frames of the scene; and

in each of multiple iterations, (i) generating a current estimate of the scene based on the multiple image frames of the scene and a prior estimate of the scene and (ii) regularizing the generated current estimate of the scene, the regularized current estimate of the scene from one iteration representing the prior estimate of the scene in a subsequent iteration;

wherein the iterations continue until the estimates of the scene converge on the higher-resolution image of the scene; and

wherein generating the initial estimate of the scene comprises warping at least some of the multiple image frames to align the image frames.

2. The method of claim 1 , wherein:

the multiple image frames of the scene comprise processed image frames generated by an image signal processor;

generating the initial estimate of the scene comprises generating the initial estimate of the scene using the processed image frames; and

generating the current estimate of the scene comprises generating the current estimate of the scene using the processed image frames.

3. The method of claim 2 , wherein:

warping at least some of the multiple image frames comprises warping at least some of the processed image frames; and

generating the initial estimate of the scene comprises:

upsampling the warped processed image frames; and

blending the upsampled warped processed image frames.

4. The method of claim 2 , wherein generating the current estimate of the scene comprises one of:

performing an inversion of the processed image frames in a Fourier domain; and

approximating an infinite impulse response (IIR) deconvolution filter using a finite impulse response (FIR) filter and applying the FIR filter to the inversion of the processed image frames.

5. The method of claim 1 , wherein:

the multiple image frames of the scene comprise raw image frames generated by one or more image sensors;

generating the initial estimate of the scene comprises generating the initial estimate of the scene using the raw image frames; and

generating the current estimate of the scene comprises generating the current estimate of the scene using the raw image frames.

6. The method of claim 5 , wherein generating the initial estimate of the scene comprises:

pre-processing at least some of the raw image frames, wherein warping at least some of the multiple image frames comprises warping at least some of the pre-processed raw image frames;

upsampling the warped pre-processed raw image frames; and

blending the upsampled warped pre-processed raw image frames.

7. The method of claim 5 , wherein generating the current estimate of the scene comprises performing an inversion of the raw image frames using a projected conjugate gradient technique or a conjugate gradient steepest descent technique.

8. An electronic device comprising:

at least one camera; and

at least one processing device configured to obtain multiple image frames of a scene and process the multiple image frames of the scene to generate a higher-resolution image of the scene;

wherein, to process the multiple image frames of the scene, the at least one processing device is configured to:

generate an initial estimate of the scene based on the multiple image frames of the scene;

in each of multiple iterations, (i) generate a current estimate of the scene based on the multiple image frames of the scene and a prior estimate of the scene and (ii) regularize the generated current estimate of the scene, the regularized current estimate of the scene from one iteration representing the prior estimate of the scene in a subsequent iteration; and

continue the iterations until the estimates of the scene converge on the higher-resolution image of the scene; and

wherein, to generate the initial estimate of the scene, the at least one processing device is configured to warp at least some of the multiple image frames to align the image frames.

9. The electronic device of claim 8 , wherein:

the multiple image frames of the scene comprise processed image frames generated by an image signal processor;

the at least one processing device is configured to generate the initial estimate of the scene using the processed image frames; and

the at least one processing device is configured to generate the current estimate of the scene using the processed image frames.

10. The electronic device of claim 9 , wherein:

to warp at least some of the multiple image frames, the at least one processing device is configured to warp at least some of the processed image frames; and

to generate the initial estimate of the scene, the at least one processing device is configured to:

upsample the warped processed image frames; and

blend the upsampled warped processed image frames.

11. The electronic device of claim 9 , wherein, to generate the current estimate of the scene, the at least one processing device is configured to one of:

perform an inversion of the processed image frames in a Fourier domain; and

approximate an infinite impulse response (IIR) deconvolution filter using a finite impulse response (FIR) filter and apply the FIR filter to the inversion of the processed image frames.

12. The electronic device of claim 8 , wherein:

the multiple image frames of the scene comprise raw image frames generated by one or more image sensors;

the at least one processing device is configured to generate the initial estimate of the scene using the raw image frames; and

the at least one processing device is configured to generate the current estimate of the scene using the raw image frames.

13. The electronic device of claim 12 , wherein, to generate the initial estimate of the scene, the at least one processing device is configured to:

pre-process at least some of the raw image frames, wherein, to warp at least some of the multiple image frames, the at least one processing device is configured to warp at least some of the pre-processed raw image frames;

upsample the warped pre-processed raw image frames; and

blend the upsampled warped pre-processed raw image frames.

14. The electronic device of claim 12 , wherein, to generate the current estimate of the scene, the at least one processing device is configured to perform an inversion of the raw image frames using a projected conjugate gradient technique or a conjugate gradient steepest descent technique.

15. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

obtain multiple image frames of a scene using at least one camera of the electronic device; and

process the multiple image frames of the scene to generate a higher-resolution image of the scene;

wherein the instructions that when executed cause the at least one processor to process the multiple image frames of the scene comprise instructions that when executed cause the at least one processor to:

generate an initial estimate of the scene based on the multiple image frames of the scene;

in each of multiple iterations, (i) generate a current estimate of the scene based on the multiple image frames of the scene and a prior estimate of the scene and (ii) regularize the generated current estimate of the scene, the regularized current estimate of the scene from one iteration representing the prior estimate of the scene in a subsequent iteration; and

continue the iterations until the estimates of the scene converge on the higher-resolution image of the scene; and

wherein the instructions that when executed cause the at least one processor to generate the initial estimate of the scene comprise instructions that when executed cause the at least one processor to warp at least some of the multiple image frames to align the image frames.

16. The non-transitory machine-readable medium of claim 15 , wherein:

the multiple image frames of the scene comprise processed image frames generated by an image signal processor;

the instructions that when executed cause the at least one processor to generate the initial estimate of the scene comprise instructions that when executed cause the at least one processor to generate the initial estimate of the scene using the processed image frames; and

the instructions that when executed cause the at least one processor to generate the current estimate of the scene comprise instructions that when executed cause the at least one processor to generate the current estimate of the scene using the processed image frames.

17. The non-transitory machine-readable medium of claim 16 , wherein:

the instructions that when executed cause the at least one processor to warp at least some of the multiple image frames comprise instructions that when executed cause the at least one processor to warp at least some of the processed image frames; and

the instructions that when executed cause the at least one processor to generate the initial estimate of the scene comprise instructions that when executed cause the at least one processor to:

upsample the warped processed image frames; and

blend the upsampled warped processed image frames.

18. The non-transitory machine-readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to generate the current estimate of the scene comprise instructions that when executed cause the at least one processor to one of:

perform an inversion of the processed image frames in a Fourier domain; and

approximate an infinite impulse response (IIR) deconvolution filter using a finite impulse response (FIR) filter and apply the FIR filter to the inversion of the processed image frames.

19. The non-transitory machine-readable medium of claim 15 , wherein:

the multiple image frames of the scene comprise raw image frames generated by one or more image sensors;

the instructions that when executed cause the at least one processor to generate the initial estimate of the scene comprise instructions that when executed cause the at least one processor to generate the initial estimate of the scene using the raw image frames; and

the instructions that when executed cause the at least one processor to generate the current estimate of the scene comprise instructions that when executed cause the at least one processor to generate the current estimate of the scene using the raw image frames.

20. The non-transitory machine-readable medium of claim 19 , wherein:

the instructions that when executed cause the at least one processor to generate the initial estimate of the scene comprise instructions that when executed cause the at least one processor to:

pre-process at least some of the raw image frames, wherein the instructions that when executed cause the at least one processor to warp at least some of the multiple image frames comprise instructions that when executed cause the at least one processor to warp at least some of the pre-processed raw image frames;

upsample the warped pre-processed raw image frames; and

blend the upsampled warped pre-processed raw image frames; and

the instructions that when executed cause the at least one processor to generate the current estimate of the scene comprise instructions that when executed cause the at least one processor to perform an inversion of the raw image frames using a projected conjugate gradient technique or a conjugate gradient steepest descent technique.