Image processing apparatus and method, and image capturing apparatus and control method thereof, and storage medium
An image processing apparatus comprises: an acquisition unit that acquires a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquires at least one parallax image pair having parallax by pupil division; a first generator that generates a first distance image from the parallax image pair; a second generator that generates a second distance image from the plurality of different viewpoint images; and an integrator that integrates the first distance image and the second distance image and generates an integrated distance image.
1. An image processing apparatus comprising one or more processors and/or circuitry which functions as:
an acquisition unit that acquires a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquires at least one parallax image pair having parallax by pupil division;
a first generator that generates a first distance image from the parallax image pair;
a second generator that generates a second distance image from the plurality of different viewpoint images; and
an integrator that integrates the first distance image and the second distance image and generates an integrated distance image,
wherein the integrator integrates the first distance image and the second distance image by complementing distance information of an area where a subject is moving among distance information constituting the second distance image with distance information of a same area in the first distance image.
2. The image processing apparatus according to claim 1 , wherein the integrator integrates the first distance image and the second distance image by complementing distance information of an area excluding an area indicating a distance shorter than a predetermined distance, and an area indicating a predetermined range from an in-focus distance among distance information constituting the first distance image with distance information of a same area in the second distance image.
3. The image processing apparatus according to claim 1 , wherein the second generator generates the second distance image based on parallax between the plurality of different viewpoint images.
4. The image processing apparatus according to claim 3 , wherein the second generator generates the second distance image using epipolar geometry based on the plurality of different viewpoint images.
5. The image processing apparatus according to claim 1 , wherein the acquisition unit acquires a plurality of parallax image pairs obtained by shooting a same scene at different positions, and
wherein the image processing apparatus further comprises a third generator that generates a plurality of different viewpoint images from the plurality of parallax image pairs.
6. The image processing apparatus according to claim 5 , wherein the third generator selects one of the plurality of parallax image pairs as a reference parallax image pair and generates the plurality of different viewpoint images by adding images constituting each of the plurality of parallax image pairs except for the reference parallax image pair pixel by pixel.
7. The image processing apparatus according to claim 5 , wherein the third generator keeps one of the plurality of parallax image pairs as a reference parallax image pair as it is, and generates the plurality of different viewpoint images by adding images constituting each of the plurality of parallax image pairs including the reference parallax image pair pixel by pixel.
8. The image processing apparatus according to claim 6 , wherein the third generator sequentially shifts selection of the reference parallax image pair among the plurality of parallax image pairs and generates the plurality of different viewpoint images for each of the sequentially selected reference parallax image pairs, and
wherein the integrator integrates, for each of the sequentially selected reference parallax image pairs, the first distance image generated from the reference parallax image pair and the second distance image generated from the plurality of different viewpoint images corresponding to the reference parallax image pair, and generates a plurality of the integrated distance images.
9. The image processing apparatus according to claim 7 , wherein the third generator sequentially shifts selection of the reference parallax image pair among the plurality of parallax image pairs and generates the plurality of different viewpoint images for each of the sequentially selected reference parallax image pairs, and
wherein the integrator integrates, for each of the sequentially selected reference parallax image pairs, the first distance image generated from the reference parallax image pair and the second distance image generated from the plurality of different viewpoint images corresponding to the reference parallax image pair, and generates a plurality of the integrated distance images.
10. The image processing apparatus according to claim 5 , wherein the third generator performs defocus deblurring processing on the plurality of parallax image pairs and generates the plurality of different viewpoint images from the plurality of parallax image pairs undergone the defocus deblurring processing.
11. The image processing apparatus according to claim 10 , wherein the defocus deblurring processing includes deconvolution processing or maximum a posteriori (MAP) estimation processing performed by estimating defocus kernel, and deep learning processing by end-to-end processing using an encoder-decoder structure.
12. The image processing apparatus according to claim 1 , wherein the acquisition unit acquires a plurality of parallax image pairs obtained by shooting a same scene at different positions,
wherein the first generator generates the first distance image from each of a plurality of predetermined parallax image pairs among the plurality of parallax image pairs, selects one of a plurality of the generated first distance images as a reference first distance image, changes viewpoints of the first distance images other than the reference first distance image, and integrates the first distance images whose viewpoints are changed with the reference first distance image, and
wherein the integrator integrates the integrated first distance image and the second distance image.
13. An image capturing apparatus comprising:
(A) an image processing apparatus comprising one or more processors and/or circuitry which functions as: (1) an acquisition unit that acquires a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquires at least one parallax image pair having parallax by pupil division; (2) a first generator that generates a first distance image from the parallax image pair; (3) a second generator that generates a second distance image from the plurality of different viewpoint images; and (4) an integrator that integrates the first distance image and the second distance image and generates an integrated distance image; and
(B) an image sensor that shoots the at least one parallax image pair at least as part of the acquisition unit,
wherein the integrator integrates the first distance image and the second distance image by complementing distance information of an area where a subject is moving among distance information constituting the second distance image with distance information of a same area in the first distance image.
14. The image capturing apparatus according to claim 13 , wherein the image sensor shoots the at least one parallax image pair after shooting the plurality of different viewpoint images.
15. The image capturing apparatus according to claim 13 , wherein the image sensor shoots the plurality of different viewpoint images after shooting the at least one parallax image pair.
16. The image capturing apparatus according to claim 13 , wherein the image sensor shoots the plurality of different viewpoint images before and after shooting the at least one parallax image pair.
17. The image capturing apparatus according to claim 13 , wherein the image sensor controls an aperture of a diaphragm for shooting the plurality of different viewpoint images to be smaller than that for shooting the at least one parallax image pairs.
18. The image capturing apparatus according to claim 13 , wherein the image sensor independently sets shooting conditions for shooting the at least one parallax image pairs and for shooting the plurality of different viewpoint images.
19. An image processing method comprising:
acquiring a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquiring at least one parallax image pair having parallax by pupil division;
generating a first distance image from the parallax image pair;
generating a second distance image from the plurality of different viewpoint images; and
integrating the first distance image and the second distance image to generate an integrated distance image,
wherein the integrating integrates the first distance image and the second distance image by complementing distance information of an area where a subject is moving among distance information constituting the second distance image with distance information of a same area in the first distance image.
20. A non-transitory computer-readable storage medium, the storage medium storing a program that is executable by a computer, wherein the program includes program code for causing the computer to execute an image processing method, the image processing method comprising:
acquiring a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquiring at least one parallax image pair having parallax by pupil division;
generating a first distance image from the parallax image pair;
generating a second distance image from the plurality of different viewpoint images; and
integrating the first distance image and the second distance image to generate an integrated distance image,
wherein the integrating integrates the first distance image and the second distance image by complementing distance information of an area where a subject is moving among distance information constituting the second distance image with distance information of a same area in the first distance image.
21. An image processing apparatus comprising one or more processors and/or circuitry which functions as:
an acquisition unit that acquires a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquires at least one parallax image pair having parallax by pupil division;
a first generator that generates a first distance image from the parallax image pair;
a second generator that generates a second distance image from the plurality of different viewpoint images; and
an integrator that integrates the first distance image and the second distance image and generates an integrated distance image,
wherein the integrator integrates the first distance image and the second distance image by complementing distance information of an area excluding an area indicating a distance shorter than a predetermined distance, and an area indicating a predetermined range from an in-focus distance among distance information constituting the first distance image with distance information of a same area in the second distance image.
22. An image processing method comprising:
acquiring a plurality of different viewpoint images obtained by shooting a same scene from different viewpoints, and acquiring at least one parallax image pair having parallax by pupil division;
generating a first distance image from the parallax image pair;
generating a second distance image from the plurality of different viewpoint images; and
integrating the first distance image and the second distance image to generate an integrated distance image,
wherein the integrating integrates the first distance image and the second distance image by complementing distance information of an area excluding an area indicating a distance shorter than a predetermined distance, and an area indicating a predetermined range from an in-focus distance among distance information constituting the first distance image with distance information of a same area in the second distance image.