Multiscale depth estimation using depth from defocus

1. A method of generating a merged physical depth map from a pair of images of a scene captured by an image capture device using different capture parameters, said method comprising:

generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and

for each pair of images:

forming a relative depth map from the pair of images;

accessing calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and

determining a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and

generating a merged physical depth map by merging, in order of increasing spatial resolution, depth values based on the defined working range associated with each of the determined physical depth maps.

2. A method according to claim 1 , wherein generating the merged physical depth map uses confidence measures associated with the relative depth estimates to weight the physical depth values from corresponding spatial resolutions when merging the physical depth values.

3. A method according to claim 1 , wherein the generating the merged physical depth map refines the physical depth map at each spatial resolution by:

upsampling a depth map of the previous coarser resolution and using the upsampled depth map as an initial refined depth map;

forming a current refined depth map by minimising an energy function that penalises depth values that, at least one of:

(i) deviate from the physical depth measurements of the current spatial resolution; and

(ii) produce large local depth gradients.

4. A method according to claim 3 wherein for (i), the energy function operates such that a data penalty increases with the associated confidence measure of the depth measurements.

5. A method according to claim 3 wherein the penalty of (ii) is reduced at pixels where a large local depth gradient is matched by a large local intensity gradient in the associated image.

6. A method according to claim 3 , further comprising modifying depth boundaries of the refined depth map by:

(i) locating pixels that are potentially lying on a depth boundary;

(ii) dilating regions of said located pixels; and

(iii) performing edge-preserved filtering on said dilated regions of said located pixels to modify the corresponding physical depth values and thereby the corresponding depth boundary.

7. A method according to claim 6 , wherein the edge-preserved filtering comprises joint bilateral filtering.

8. A method according to claim 6 wherein step (i) comprises locating pixels with an associated image and whose depth gradient exceeds a pre-defined threshold to thereby define a depth boundary.

9. A method according to claim 6 wherein step (i) comprises identifying pixels whose depth gradient exceeds a pre-defined threshold and lie on an edge on the associated image.

10. A non-transitory computer readable storage medium having a program recorded thereon, the program being executable by a processor to generate merged physical depth map from a pair of images of a scene captured by an image capture device using different capture parameters, said program comprising:

code for generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and

code, operative for each pair of images, to:

form a relative depth map from the pair of images;

access calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and

determine a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and

code for generating a merged physical depth map by merging, in order of increasing spatial resolution, depth values based on the defined working range associated with each of the determined physical depth maps.

11. A non-transitory computer readable storage medium according to claim 10 , wherein the code for generating the merged physical depth map uses confidence measures associated with the relative depth estimates to weight the physical depth values from corresponding spatial resolutions when merging the physical depth values.

12. A non-transitory computer readable storage medium according to claim 10 , wherein the code for generating the merged physical depth map refines the physical depth map at each spatial resolution by:

upsampling a depth map of the previous coarser resolution and using the upsampled depth map as an initial refined depth map;

forming a current refined depth map by minimising an energy function that penalises depth values that, at least one of:

(i) deviate from the physical depth measurements of the current spatial resolution ; and

(ii) produce large local depth gradients.

13. A non-transitory computer readable storage medium according to claim 12 wherein for (i), the energy function operates such that a data penalty increases with the associated confidence measure of the depth measurements, and the penalty of (ii) is reduced at pixels where a large local depth gradient is matched by a large local intensity gradient in the associated image.

14. A non-transitory computer readable storage medium according to claim 11 , further comprising code for modifying depth boundaries of the refined depth map by:

(i) locating pixels that are potentially lying on a depth boundary;

(ii) dilating regions of said located pixels; and

(iii) performing edge-preseved filtering on said dilated regions of said located pixels to modify the corresponding physical depth values and thereby the corresponding depth boundary.

15. A non-transitory computer readable storage medium according to claim 14 wherein step (i) comprises locating pixels with an associated image and whose depth gradient exceeds a pre-defined threshold to thereby define a depth boundary, and the edge-preserved filtering comprises joint bilateral filtering.

16. A non-transitory computer readable storage medium according to claim 14 wherein step (i) comprises identifying pixels whose depth gradient exceeds a pre-defined threshold and lie on an edge on the associated image, and the edge-preserved filtering comprises joint bilateral filtering.

17. Apparatus comprising a processor and a memory, the memory storing a program executable by the processor for generating a merged physical depth map from a pair of images of a scene captured by an image capture device using different capture parameters, the program comprising:

code for generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and

code, operative for each pair of images, to:

form a relative depth map from the pair of images;

access calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and

determine a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and

code for generating a merged physical depth map by merging, in order of increasing spatial resolution, depth values based on the defined working range associated with each of the determined physical depth maps.

18. Apparatus according to claim 17 , wherein code for generating the merged physical depth map uses confidence measures associated with the relative depth estimates to weight the physical depth values from corresponding spatial resolutions when merging the physical depth values.

19. Apparatus according to claim 17 , wherein the code for generating the merged physical depth map refines the physical depth map at each spatial resolution by:

upsampling a depth map of the previous coarser resolution and using the upsampled depth map as an initial refined depth map;

forming a current refined depth map by minimising an energy function that penalises depth values that, at least one of:

(i) deviate from the physical depth measurements of the current spatial resolution; and

(ii) produce large local depth gradients.

20. A camera comprising:

an image capture arrangement including a lens and an image sensor via which objects in a scene can be imaged;

a processor; and

a memory storing a program executable by the processor for generating a merged physical depth map from a pair of images of the scene captured by the image capture arrangement using different capture parameters, the program comprising:

code for generating, from the captured pair of images, at least one pair of downsampled images at a corresponding spatial resolution; and

code, operative for each pair of images, to:

form a relative depth map from the pair of images;

access calibration data of the image capture device associated with the pair of images, the calibration data relating relative depth measurements with physical depth and having a defined working range where a single relative depth measurement relates to a single physical depth; and

determine a physical depth map for the pair of the images from the formed relative depth map and the accessed calibration data; and

code for generating a merged physical depth map by merging, in order of increasing spatial resolution, depth values based on within the defined working range associated with each of the determined physical depth maps.

21. A method according to claim 2 , wherein a confidence measure associated with a relative depth estimate is determined using a working range for a corresponding calibration data.

22. A non-transitory computer readable storage medium according to claim 11 , wherein a confidence measure associated with a relative depth estimate is determined using a working range for a corresponding calibration data.

23. Apparatus according to claim 18 , wherein a confidence measure associated with a relative depth estimate is determined using a working range for a corresponding calibration data.