Method, apparatus and computer program product for image refocusing for light-field images

1. A method comprising:

determining shifts between images of a plurality of light-field images captured in a burst capture by a light-field camera, the shifts between the images of the plurality of light-field images being associated with shake of the light-field camera while capturing the plurality of light-field images;

generating a plurality of depth maps for the plurality of light-field images, wherein a depth map of the plurality of depth maps is generated for a corresponding light-field image of the plurality of light-field images;

generating a set of view images of the scene based on the plurality of light-field images and the plurality of depth maps, wherein the set of view images includes at least a first view image generated based on a first of the light-field images and a second view image generated based on a second of the light-field images; and

generating a refocus image by combining the first view image and the second view image, the second view image having been shifted based on a shift between the first view image and a non-shifted version of the second view image, wherein the shift between the first view image and the non-shifted version of the second view image is dependent on the determined shift between the first light-field image and the second light-field image.

2. The method as claimed in claim 1 , wherein determining the shifts comprises:

determining positions of the light-field camera based on one or more positional measurements of the light-field camera at instants of capturing the plurality of light-field images; and

calculating the shifts between the images of the plurality of light-field images, based on the changes in the positions of the light-field camera at the instants of capturing the images of the plurality of light-field images.

3. The method as claimed in claim 2 , wherein the one or more positional measurements are performed by a gyroscope.

4. The method as claimed in claim 1 , wherein determining the shifts comprises:

selecting a reference light-field image from the plurality of light-field images;

performing image registration of the plurality of light-field images with the reference light-field image; and

determining the shifts between the images of the plurality of light-field images based on the registration of the images of the plurality of light-field images with the reference light-field image.

5. The method as claimed in claim 1 , further comprising generating a refined depth map based at least on the plurality of images and the set of images.

6. The method as claimed in claim 1 , wherein the set of view images comprises one or more groups of view images, wherein each group of view images of the one or more groups of view images is associated with a light-field image of the plurality of light-field images.

7. The method as claimed in claim 1 , wherein generating the refocus image comprises:

selecting a view image from the set of view images;

shifting the one or more view images of the set of view images corresponding to respective shifts of the one or more view images with respect to the selected view image; and

adding the one or more shifted view images to the selected view image to generate the refocus image.

8. The method as claimed in claim 1 , wherein facilitating the burst capture of the plurality of images comprises storing a plurality of light-field image data, wherein each light-field image data comprises a plurality of sub-images captured through a plurality of micro-lenses of a micro-lens array of the light-field camera.

9. The method as claimed in claim 1 , wherein the shift between the first view image and the non-shifted version of the second view image is the determined shift between the first light-field image and the second light-field image plus a shift between the non-shifted version of the second view image and a further view image generated based on the second light-field image.

10. The method as claimed in claim 1 , wherein generating the set of view images comprises generating the set of views by concatenating at least one corner pixel of one or more sub-images of the plurality of light-field images.

11. An apparatus comprising:

at least one processor; and

at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

determine shifts between images of a plurality of light-field images captured in a burst capture by a light-field camera, the shifts between the images of the plurality of light-field images being associated with shake of the light-field camera while capturing the plurality of light-field images;

generate a plurality of depth maps for the plurality of light-field images, wherein a depth map of the plurality of depth maps is generated for a corresponding light-field image of the plurality of light-field images;

generate a set of view images of the scene based on the plurality of light-field images and the plurality of depth maps, wherein the set of view images includes at least a first view image generated based on a first of the light-field images and a second view image generated based on a second of the light-field; and

generate a refocus image by combining the first view image and the second view image, the second view image having been shifted based on a shift between the first view image and a non-shifted version of the second view image, wherein the shift between the first view image and the non-shifted version of the second view image is dependent on the determined shift between the first light-field image and the second light-field image.

12. The apparatus as claimed in claim 11 , wherein the apparatus is further caused, at least in part to:

determine positions of the light-field camera based on one or more positional measurements of the light-field camera at instants of capturing the plurality of light-field images; and

calculate the shifts between the images of the plurality of light-field images, based on the changes in the positions of the light-field camera at the instants of capturing the images of the plurality of light-field images.

13. The apparatus as claimed in claim 12 , wherein the one or more positional measurements are performed by a gyroscope.

14. The apparatus as claimed in claim 11 , wherein for determining the shifts the apparatus is further caused, at least in part to:

select a reference light-field image from the plurality of light-field images;

perform image registration of the plurality of light-field images with the reference light-field image; and

determine the shifts between the images of the plurality of light-field images based on the registration of the images of the plurality of light-field images with the reference light-field image.

15. The apparatus as claimed in claim 11 , wherein the apparatus is further caused, at least in part to generate a refined depth map based at least on the plurality of images and the set of images.

16. The apparatus as claimed in claim 11 , wherein the set of view images comprises one or more groups of view images, wherein each group of view images of the one or more groups of view images is associated with a light-field image of the plurality of light-field images.

17. The apparatus as claimed in claim 11 , wherein for generating, the apparatus is further caused, at least in part to:

selecting a view image from the set of view images;

shifting the one or more view images of the set of view images corresponding to respective shifts of the one or more view images with respect to the selected view image; and

adding the one or more shifted view images to the selected view image to generate the refocus image.

18. The apparatus as claimed in claim 11 , wherein for facilitating the burst capture, the apparatus is further caused, at least in part to:

store a plurality of light-field image data, wherein each light-field image data comprises a plurality of sub-images captured through a plurality of micro-lenses of a micro-lens array of the light-field camera.

19. The apparatus as claimed in claim 11 , wherein the shift between the first view image and the non-shifted version of the second view image is the determined shift between the first light-field image and the second light-field image plus a shift between the non-shifted version of the second view image and a further view image generated based on the second light-field image.

20. The apparatus as claimed in claim 11 , wherein generating the set of view images comprises generating the set of views by concatenating at least one corner pixel of one or more sub-images of the plurality of light-field images.

21. A computer program product comprising at least one non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a set of instructions, which, when executed by one or more processors, cause an apparatus to at least perform:

determine shifts between images of a plurality of light-field images captured in a burst capture by a light-field camera, the shifts between the images of the plurality of light-field images being associated with shake of the light-field camera while capturing the plurality of light-field images;

generate a plurality of depth maps for the plurality of light-field images, wherein a depth map of the plurality of depth maps is generated for a corresponding light-field image of the plurality of light-field images;

generate a set of view images of the scene based on the plurality of light-field images and the plurality of depth maps, wherein the set of view images includes at least a first view image generated based on a first of the light-field images and a second view image generated based on a second of the light-field images; and

generate a refocus image by combining the first view image and the second view image, the second view image having been shifted based on a shift between the first view image and a non-shifted version of the second view image, wherein the shift between the first view image and the non-shifted version of the second view image is dependent on the determined shift between the first light-field image and the second light-field image.

22. The computer program product as claimed in claim 21 , wherein the apparatus is further caused, at least in part to:

determine positions of the light-field camera based on one or more positional measurements of the light-field camera at instants of capturing the plurality of light-field images; and

calculate the shifts between the images of the plurality of light-field images, based on the changes in the positions of the light-field camera at the instants of capturing the images of the plurality of light-field images.

23. The computer program product as claimed in claim 21 , wherein for determining the shifts the apparatus is further caused, at least in part to:

select a reference light-field image from the plurality of light-field images;

perform image registration of the plurality of light-field images with the reference light-field image; and

determine the shifts between the images of the plurality of light-field images based on the registration of the images of the plurality of light-field images with the reference light-field image.

24. The computer program product as claimed in claim 21 , wherein for generating, the apparatus is further caused, at least in part to:

select a view image from the set of view images;

shifting the one or more view images of the set of view images corresponding to respective shifts of the one or more view images with respect to the selected view image; and

adding the one or more shifted view images to the selected view image to generate the refocus image.

25. The computer program product as claimed in claim 21 , wherein the shift between the first view image and the non-shifted version of the second view image is the determined shift between the first light-field image and the second light-field image plus a shift between the non-shifted version of the second view image and a further view image generated based on the second light-field image.

26. The computer program product as claimed in claim 21 , wherein generating the set of view images comprises generating the set of views by concatenating at least one corner pixel of one or more sub-images of the plurality of light-field images.