Image processing apparatus, image processing method, and imaging apparatus

1. An image processing apparatus comprising:

an image acquisition unit configured to acquire an image;

a depth map acquisition unit configured to acquire a depth map corresponding to the image;

a refinement unit configured to detect a saliency region, which is a region on which a person tends to focus, from the image and to refine the depth map on the basis of the saliency region; and

an image processing unit configured to apply image processing to the image using the depth map refined by the refinement unit,

wherein the image processing unit is configured to divide depths in the depth map into a plurality of depth layers, and the refinement unit is configured to judge whether the saliency region includes depths in two or more depth layers and to refine the depths in the saliency region in the depth map to be smooth when the saliency region is judged to include two or more depth layers.

2. The image processing apparatus according to claim 1 , wherein

the depth map acquisition unit is further configured to acquire a reliability of depth calculation in the depth map, and

the refinement unit is configured to refine a depth of a part of the depth map, the part of the depth map being within the saliency region and having the reliability equal to or smaller than a threshold.

3. The image processing apparatus according to claim 1 , wherein the saliency region is a region including a character.

4. The image processing apparatus according to claim 1 , wherein the saliency region is a region including a straight line.

5. The image processing apparatus according to claim 1 , wherein the saliency region is a region with intensity of an edge component equal to or larger than a predetermined threshold.

6. The image processing apparatus according to claim 1 , wherein the image processing is blurring processing.

7. The image processing apparatus according to claim 1 , wherein the image processing is image restoration processing or super-resolution processing.

8. An imaging apparatus comprising:

the image processing apparatus according to claim 1 ; and

an image pickup unit configured to convert light from an object into an electric signal and output the electric signal, wherein

an image is acquired by the image acquisition unit on the basis of the electric signal.

9. An image processing apparatus comprising:

an image acquisition unit configured to acquire an image;

a depth map acquisition unit configured to acquire a depth map corresponding to the image and to acquire a reliability of depth calculation in the depth map;

a refinement unit configured to detect a saliency region, which is a region on which a person tends to focus, from the image and to refine the depth map on the basis of the saliency region; and

an image processing unit configured to apply image processing to the image using the depth map refined by the refinement unit,

wherein the refinement unit is configured to refine a depth of a part of the depth map, the part of the depth map being within the saliency region and having the reliability equal to or smaller than a threshold, and to calculate the depth in the part of the depth map by interpolation on the basis of a depth around the part of the depth map.

10. An image processing apparatus comprising:

an image acquisition unit configured to acquire an image;

a depth map acquisition unit configured to acquire a depth map corresponding to the image and to acquire reliability of depth calculation in the depth map;

a refinement unit configured to detect a saliency region, which is a region on which a person tends to focus, from the image and to refine the depth map on the basis of the saliency region; and

an image processing unit configured to apply image processing to the image using the depth map refined by the refinement unit,

wherein the refinement unit is configured to refine a depth of a part of the depth map, the part of the depth map being within the saliency region and having the reliability equal to or smaller than a threshold, and to set all depths in the part of the depth map into any one of the depths in the part of the depth map.

11. The image processing apparatus according to claim 10 , wherein the refinement unit is configured to set all the depths in the saliency region into a depth corresponding to a most predominant depth layer among the depth layers included in the saliency region.

12. An image processing method performed by an image processing apparatus, the image processing method comprising:

an image acquisition step of acquiring an image;

a depth map acquisition step of acquiring a depth map;

a detecting step of detecting a saliency region, which is a region on which a person tends to focus, from the image;

a refinement step of refining the depth map on the basis of the saliency region; and

an image processing step of applying image processing to the image using the depth map refined in the refinement step,

wherein the image processing step divides depths in the depth map into a plurality of depth layers, and the refinement step judges whether the saliency region includes depths in two or more depth layers and refines the depths in the saliency region in the depth map to be smooth when the saliency region is judged to include two or more depth layers.

13. The image processing method according to claim 12 , wherein

in the depth map acquisition step, reliability of depth calculation in the depth map is acquired as well, and

in the refinement step, a depth of a part of the depth map is refined, the part of the depth map being a region within the saliency region and having the reliability equal to or smaller than a threshold.

14. The image processing method according to claim 12 , wherein the saliency region is a region including a character, a region including a straight line, or a region with intensity of an edge component equal to or larger than a predetermined threshold.

15. The image processing method according to claim 12 , wherein the image processing is blurring processing, image restoration processing, or super-resolution processing.

16. An image processing method performed by an image processing apparatus, the image processing method comprising:

an image acquisition step of acquiring an image;

a depth map acquisition step of acquiring a depth map corresponding to the image and acquiring a reliability of depth calculation in the depth map;

a refinement step of detecting a saliency region, that is a region on which a person tends to focus, from the image, and refining the depth map on the basis of the saliency region; and

an image processing step of applying image processing to the image using the depth map refined in the refinement step,

wherein the refinement step refines a depth of a part of the depth map, the part of the depth map being within the saliency region and having a reliability equal to or smaller than a threshold, and all depths in the part of the depth map are set into any one of the depths in the part of the depth map.

17. An image processing method performed by an image processing apparatus, the image processing method comprising:

an image acquisition step of acquiring an image;

a depth map acquisition step of acquiring a depth map corresponding to the image and acquiring a reliability of depth calculation for the depth map;

a refinement step of detecting a saliency region, which is a region on which a person tends to focus, from the image and refining the depth map on the basis of the saliency region, wherein a depth of a part of the depth map that is within the saliency region and that has a reliability equal to or smaller than a threshold is refined; and

an image processing step of applying image processing to the image using the depth map refined in the refinement step,

wherein the refinement step calculates the depth in the part of the depth map by interpolation on the basis of a depth around the part of the depth map.

18. A non-transitory computer readable storage medium storing a program which causes a computer to perform an image processing method, the image processing method comprising:

an image acquisition step of acquiring an image;

a depth map acquisition step of acquiring a depth map;

a detecting step of detecting a saliency region, which is a region on which a person tends to focus;

a refinement step of refining the depth map on the basis of the saliency region; and

an image processing step of applying image processing to the image using the depth map refined in the refinement step,

wherein the image processing step divides depths in the depth map into a plurality of depth layers, and the refinement step judges whether the saliency region includes depths in two or more depth layers and to refine the depths in the saliency region in the depth map to be smooth when the saliency region is judged to include two or more depth layers.

19. A non-transitory computer readable storage medium storing a program which causes a computer to perform an image processing method, the image processing method comprising:

an image acquisition step of acquiring an image;

a depth map acquisition step of acquiring a depth map corresponding to the image and acquiring a reliability of depth calculation in the depth map;

a refinement step of detecting a saliency region, that is a region on which a person tends to focus, from the image, and refining the depth map on the basis of the saliency region; and

an image processing step of applying image processing to the image using the depth map refined in the refinement step,

wherein the refinement step refines a depth of a part of the depth map, the part of the depth map being within the saliency region and having a reliability equal to or smaller than a threshold, and all depths in the part of the depth map are set into any one of the depths in the part of the depth map.

20. A non-transitory computer readable storage medium storing a program which causes a computer to perform an image processing method, the image processing method comprising:

an image acquisition step of acquiring an image;

a depth map acquisition step of acquiring a depth map corresponding to the image and acquiring a reliability of depth calculation for the depth map;

a refinement step of detecting a saliency region, which is a region on which a person tends to focus, from the image and refining the depth map on the basis of the saliency region, wherein a depth of a part of the depth map that is within the saliency region and that has a reliability equal to or smaller than a threshold is refined; and

an image processing step of applying image processing to the image using the depth map refined in the refinement step,

wherein the refinement step calculates the depth in the part of the depth map by interpolation on the basis of a depth around the part of the depth map.