Image processing method, image processing device, and image pickup apparatus

1. An image processing method comprising:

acquiring a plurality of viewpoint images;

generating a contrast distribution from the plurality of viewpoint images;

generating, from the plurality of viewpoint images, one of (a) an image shift amount distribution indicating a distribution of an image shift amount between viewpoint images, of the plurality of viewpoint images, and (b) a defocus amount distribution indicating a distribution of a defocus amount of an object in the viewpoint images, of the plurality of viewpoint images; and

generating an output image by performing image processing in accordance with the contrast distribution with respect to an image based on the plurality of viewpoint images, the image processing being one of processing for sharpening, processing for smoothing, and a combined processing of processing for sharpening and processing for smoothing, and the image processing being performed based on one of (a) the image shift amount distribution and the contrast distribution, and (b) the defocus amount distribution and the contrast distribution.

2. The image processing method according to claim 1 , wherein the contrast distribution is generated in accordance with a difference between a contrast of each viewpoint image, of the plurality of viewpoint images, with other viewpoint images, of the plurality of viewpoint images.

3. The image processing method according to claim 1 , wherein the image processing is performed at a greater intensity in an area with a relatively lower difference between the contrast in each viewpoint image, of the plurality of viewpoint images, than in another area with a relatively greater difference therebetween.

4. The image processing method according to claim 1 , wherein processing for sharpening is performed, as the image processing, at a greater intensity in an area with one of (a) a relatively lower difference of the image shift amount distribution than in another area with a relatively greater difference of the image shift amount distribution, and (b) a relatively lower difference of the defocus amount distribution than in another area with a relatively greater difference of the defocus amount distribution.

5. The image processing method according to claim 1 , wherein processing for smoothing is performed, as the image processing, at a greater intensity in an area with one of (a) a relatively greater difference of the image shift amount distribution than in another area with a relatively lower difference of the image shift about distribution, and (b) a relatively greater difference of the defocus amount distribution than in an area with a relatively lower difference of the defocus amount distribution.

6. The image processing method according to claim 1 , wherein processing for sharpening is performed, as the image processing, in an area with one of (a) a relatively lower difference of the image shift amount distribution, and (b) a relatively lower difference of the defocus amount distribution, and processing for smoothing is performed, as the image processing, in another area with one of (a) a relatively greater difference of the image shift amount distribution, and (b) a relatively greater difference of the defocus amount distribution.

7. The image processing method according to claim 1 , wherein the image processing is processing for modifying saturation.

8. The image processing method according to claim 1 , wherein, in a synthesis image made by synthesizing the plurality of viewpoint images, a synthesis contrast distribution is generated to be greater in an area with a greater amount of space frequency components in a predetermined space frequency band than in another area having a lesser amount of space frequency components in the predetermined space frequency band.

9. The image processing method according to claim 8 , wherein the synthesis image is generated by multiplying a weighting coefficient by each viewpoint image, of the plurality of viewpoint images.

10. The image processing method according to claim 1 , wherein, in a synthesis image made by synthesizing the plurality of viewpoint images, the contrast distribution is greater in an area of the synthesis image with a high brightness than in another area of the synthesis image with a low brightness.

11. The image processing method according to claim 1 , wherein the contrast distribution is greater in an area with a relatively lower difference between the contrast of each viewpoint image, of the plurality of viewpoint images, than in another area with a relatively greater difference between the contrast of each viewpoint image, of the plurality of viewpoint images.

12. The image processing method according to claim 1 , wherein the image processing is performed at a greater intensity in an area with a relatively greater contrast distribution than in another area with a relatively lower contrast distribution.

13. The image processing method according to claim 1 , wherein the contrast distribution is stored in a storage medium.

14. The image processing method according to claim 1 , wherein the image shift amount distribution is generated in an area of each viewpoint image, of the plurality of viewpoint images, in which the contrast distribution is greater than or equal to a predetermined value.

15. The image processing method according to claim 1 , wherein the image shift amount distribution is stored in a storage medium.

16. The image processing method according to claim 1 , wherein the plurality of viewpoint images is generated from an input image acquired by an imaging sensor, in which pixels, of a plurality of pixels provided with a plurality of photoelectric conversion units for receiving light flux passing through different partial pupil areas of the image sensor, are arrayed.

17. An image processing device comprising:

(A) at least one memory that stores instructions;

(B) at least one processor coupled to the memory and configured to execute the instructions to function as:

(a) an acquiring unit configured to acquire a plurality of viewpoint images;

(b) a contrast distribution generating unit configured to generate a contrast distribution from the plurality of viewpoint images;

(c) a generating unit configured to generate, from the plurality of viewpoint images, one of (i) an image shift amount distribution indicating a distribution of an image shift amount between viewpoint images, of the plurality of viewpoint images, and (ii) a defocus amount distribution indicating a distribution of a defocus amount of an object in the view point images, of the plurality of viewpoint images; and

(d) an output image generating unit configured to generate an output image by performing image processing in accordance with the contrast distribution with respect to an image based on the plurality of viewpoint images, the image processing being one of processing for sharpening, processing for smoothing, and a combined processing of processing for sharpening and processing for smoothing, and the image processing being performed based on one of (i) the image shift amount distribution and the contrast distribution, and (ii) the defocus amount distribution and the contrast distribution.

18. An image pickup apparatus comprising:

(A) an imaging sensor, in which pixels, of a plurality of pixels provided with a plurality of sub pixels for receiving light flux passing through different partial pupil areas in the imaging sensor, are arrayed, and configured to acquire an input image;

(B) at least one memory that stores instructions; and

(C) at least one processor coupled to the at least one memory and configured to execute the instructions to function as:

(a) a viewpoint image generating unit configured to generate a plurality of viewpoint images in every area of the different partial pupil areas from the input image acquired from the imaging sensor;

(b) a captured image generating unit configured to generate a captured image in accordance with a pupil area, in which the different partial pupil areas are synthesized, from the input image;

(c) a contrast distribution generating unit configured to generate a contrast distribution from the plurality of viewpoint images;

(d) a distribution generating unit configured to generate, from the plurality of viewpoint images, one of (i) an image shift amount distribution indicating a distribution of an image shift amount between viewpoint images, of the plurality of viewpoint images, and (ii) a defocus amount distribution indicating a distribution of a defocus amount of an object in the viewpoint images, of the plurality of viewpoint images; and

(e) an output image generating unit configured to generate an output image by performing image processing in accordance with the contrast distribution, with respect to an image, based on the plurality of viewpoint images, the image processing being one of processing for sharpening, processing for smoothing, and a combined processing of processing for sharpening and processing for smoothing, and the image processing being performed based on one of (i) the image shift amount distribution and the contrast distribution, and (ii) the defocus amount distribution and the contrast distribution.