System and method for providing dolly zoom view synthesis

1. A method of providing a dolly zoom effect by an electronic device, the method comprising:

obtaining a first image with a corresponding first depth map and a second image with a corresponding second depth map, wherein the first image and the second image have different fields of view;

generating a first synthesized image and a corresponding first synthesized depth map using the first image and the first depth map respectively;

generating a second synthesized image and a corresponding second synthesized depth map using the second image and the second depth map respectively;

generating a fused image from the first synthesized image and the second synthesized image;

generating a fused depth map from the first synthesized depth map and the second synthesized depth map;

performing hole-filling on the fused depth map and the fused image; and

after performing hole-filling, applying a shallow depth of field effect to the fused depth map and the fused image to obtain a third synthesized image with the dolly zoom effect.

2. The method of claim 1 , wherein obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map comprises:

receiving the first image and a third image from a first camera;

determining the first depth map for the first image by performing depth estimation using the first image and the third image;

determining the second image by performing a digital zoom on the first image; and

determining the second depth map by performing a digital zoom on the first depth map.

3. The method of claim 1 , wherein obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map comprises:

receiving the first image and the second image from a first camera;

receiving a third image from a second camera;

determining the first depth map by performing depth estimation using the first image and the second image; and

determining the second depth map by performing depth estimation using the second image and the third image.

4. The method of claim 1 , wherein obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map comprises:

receiving the first image from a first camera;

deriving the first depth map;

determining the second image by performing a digital zoom on the first image; and

determining the second depth map by performing a digital zoom on the first depth map.

5. The method of claim 1 , wherein obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map comprises:

receiving the first image from a first camera;

receiving the second image from a second camera; and

deriving the first depth map and the second depth map.

6. The method of claim 1 , wherein generating the first synthesized image and the corresponding first synthesized depth map, comprises:

setting a camera translation along at least one axis for the first image;

setting a focus distance via user input; and

warping the first image and the first depth map using forward warping, based on the camera translation for the first image and the focus distance, to obtain the first synthesized image and the first synthesized depth map.

7. The method of claim 6 , wherein generating the second synthesized image and the corresponding second synthesized depth map, comprises:

setting a camera translation along the at least one axis for the second image; and

warping the second image and the second depth map using forward warping, based on the camera translation for the second image and the focus distance, to obtain the second synthesized image and the second synthesized depth map.

8. The method of claim 1 , wherein performing hole-filling comprises:

performing depth hole-filling using the fused depth map to obtain a hole-filled depth map; and

performing depth-aware image inpainting using the fused image and the hole-filled depth map to obtain a hole-filled image.

9. The method of claim 8 , wherein performing depth-aware image inpainting comprises:

updating an occlusion mask based on each of a plurality of unique depth values in the hole-filled depth map; and

applying a window averaging filter for occluded areas in the fused image according to the updated occlusion mask.

10. The method of claim 8 , wherein the shallow depth of field effect is applied using the hole-filled image and the hole-filled depth map.

11. An electronic device, comprising:

at least one camera;

a processor, which, upon instructions being executed, is caused to:

obtain a first image with a corresponding first depth map and a second image with a corresponding second depth map, wherein the first image and the second image have different fields of view;

generate a first synthesized image and a corresponding first synthesized depth map using the first image and the first depth map respectively;

generate a second synthesized image and a corresponding second synthesized depth map using the second image and the second depth map respectively;

generate a fused image from the first synthesized image and the second synthesized image;

generate a fused depth map from the first synthesized depth map and the second synthesized depth map;

perform hole-filling on the fused depth map and the fused image; and

after performing hole-filling, apply a shallow depth of field effect to the fused depth map and the fused image to obtain a third synthesized image with a dolly zoom effect.

12. The electronic device of claim 11 , wherein, in obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map, the instructions further cause the processor to:

receive the first image and a third image from a first camera;

determine the first depth map for the first image by performing depth estimation using the first image and the third image;

determine the second image by performing a digital zoom on the first image; and

determine the second depth map by performing a digital zoom on the first depth map.

13. The electronic device of claim 11 , wherein, in obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map, the instructions further cause the processor to:

receive the first image and the second image from a first camera;

receive a third image from a second camera;

determine the first depth map by performing depth estimation using the first image and the second image; and

determine the second depth map by performing depth estimation using the second image and the third image.

14. The electronic device of claim 11 , wherein, in obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map, the instructions further cause the processor to:

receive the first image from a first camera;

derive the first depth map;

determine the second image by performing a digital zoom on the first image; and

determine the second depth map by performing a digital zoom on the first depth map.

15. The electronic device of claim 11 , wherein, in obtaining the first image with the corresponding first depth map and the second image with the corresponding second depth map, the instructions further cause the processor to:

receive the first image from a first camera;

receive the second image from a second camera; and

derive the first depth map and second depth map.

16. The electronic device of claim 11 , wherein, in generating the first synthesized image and the corresponding first synthesized depth map, the instructions further cause the processor to:

set a camera translation along at least one axis for the first image;

set a focus distance via user input; and

warp the first image and the first depth map using forward warping, based on the camera translation for the first image and the focus distance, to obtain the first synthesized image and the first synthesized depth map.

17. The electronic device of claim 16 , wherein, in generating the second synthesized image and the corresponding second synthesized depth map, the instructions further cause the processor to:

set a camera translation along the at least one axis for the second image; and

warp the second image and the second depth map using forward warping, based on the camera translation for the second image and the focus distance, to obtain the second synthesized image and the second synthesized depth map.

18. The electronic device of claim 11 , wherein, in performing hole-filling, the instructions further cause the processor to:

perform depth hole-filling using the fused depth map to obtain a hole-filled depth map; and

perform depth-aware image inpainting using the fused image and the hole-filled depth map to obtain a hole-filled image.

19. The electronic device of claim 18 , wherein, in performing depth-aware image inpainting, the instructions further cause the processor to:

update an occlusion mask based on each of a plurality of unique depth values in the hole-filled depth map; and

apply a window averaging filter for occluded areas in the fused image according to the updated occlusion mask.

20. The electronic device of claim 18 , wherein the shallow depth of field effect is applied using the hole-filled image and the hole-filled depth map.