Image processing apparatus, image processing method, and program for combining the multi-viewpoint image data

1. An image processing apparatus that generates composite image data by combining multi-viewpoint image data corresponding to a plurality of viewpoints, the device comprising:

a first input unit configured to input a virtual focus distance of the composite image data;

a second input unit configured to input distance information indicative of a distance of a subject in the composite image data from a viewpoint included in the plurality of viewpoints;

a determination unit configured to input information indicative of a positional relationship of the plurality of viewpoints and determine, based on the virtual focus distance, the distance information and the positional relationship, a size of a blurring filter, to be applied to the multi-viewpoint image data, which blurs an image;

a filter processing unit configured to perform filter processing on the multi-viewpoint image data using the blurring filter of the size determined by the determination unit; and

a generation unit configured to generate composite image data by combining the multi-viewpoint image data on which the filter processing has been performed.

2. The image processing apparatus according to claim 1 , wherein

the distance information has a distance to a subject for each region, and

the filter processing unit performs filter processing on the multi-viewpoint image data based on the distance information indicative of the distance to a subject for each region.

3. The image processing apparatus according to claim 2 , wherein the region is a pixel.

4. The image processing apparatus according to claim 1 , wherein the information indicative of the positional relationships of the plurality of viewpoints indicates an interval of the plurality of viewpoints.

5. The image processing apparatus according to claim 1 , further comprising a distance information acquisition unit configured to acquire the distance information indicative of a distance to a subject.

6. The image processing apparatus according to claim 5 , wherein the distance information acquisition unit acquires the distance information indicative of a distance to a subject by calculating an optical flow between images indicated by the multi-viewpoint image data.

7. The image processing apparatus according to claim 5 , wherein the distance information acquisition unit acquires the distance information indicative of a distance to a subject using a distance measuring sensor.

8. The image processing apparatus according to claim 1 , wherein the blurring filter is a low-pass filter.

9. The image processing apparatus according to claim 1 , further comprising an specification unit configured to specify a region in focus within images captured from the plurality of viewpoints, wherein

the filter processing unit causes the blurring filter to act on a region other than the region in focus, the blurring filter concentrating more power in a region of lower frequencies than in the region in focus.

10. An image processing method for generating composite image data by combining multi-viewpoint image data corresponding to a plurality of viewpoints, the method comprising the steps of:

inputting a virtual focus distance of the composite image data;

inputting distance information indicative of a distance of a subject in the composite image data from a viewpoint included in the plurality of viewpoints;

inputting information indicative of a positional relationship of the plurality of viewpoints and determining, based on the virtual focus distance, the distance information and the positional relationship, a size of a blurring filter, to be applied to the multi-viewpoint image data, which blurs an image;

performing filter processing on the multi-viewpoint image data using the blurring filter of the size determined by the determining step; and

generating composite image data by combining the multi-viewpoint image data on which the filter processing has been performed.

11. A program stored in a non-transitory computer readable storage medium for causing a computer to perform the image processing method according to claim 10 .

12. An image processing apparatus that generates composite image data by combining multi-viewpoint image data corresponding to a plurality of viewpoints, the apparatus comprising:

a first input unit configured to input a virtual focus distance of the composite image data;

a second input unit configured to input distance information indicative of a distance of a subject included in the composite image data from a viewpoint included in the plurality of viewpoints;

a determination unit configured to determine a filter processing to blur images represented by the multi-viewpoint image data based on the virtual focus distance and the distance information;

a filter processing unit configured to perform the filter processing determined by the determination unit on the images represented by the multi-viewpoint image data; and

a generating unit configured to generate a composite image data by aligning the images represented by the multi-viewpoint image data after the filter processing and combining the images represented by the multi-viewpoint image data after the alignment,

wherein a difference between the distance of the subject and the virtual focus distance increases when a difference in a pixel position of the subject between the images represented by the multi-viewpoint image data after the alignment increases.

13. The image processing apparatus according to claim 12 , wherein a shift amount of each pixel of the images represented by the multi-viewpoint image data used for the alignment is constant for each image regardless of a position of the pixel in the image.

14. The image processing apparatus according to claim 12 , wherein the distance information includes a plurality of distance maps each of which corresponds to one of the plurality of viewpoints and indicates a distance of a subject corresponding to each pixel, and the filter processing unit is configured to perform the filter processing on each of the images represented by the multi-viewpoint image data using a corresponding distance map.

15. An image processing method for generating composite image data by combining multi-viewpoint image data corresponding to a plurality of viewpoints, the method comprising the steps of:

inputting a virtual focus distance of the composite image data;

inputting distance information indicative of a distance of a subject included in the composite image data from a viewpoint included in the plurality of viewpoints;

determining a filter processing to blur images represented by the multi-viewpoint image data based on the virtual focus distance and the distance information;

performing the filter processing determined by the determining step on the images represented by the multi-viewpoint image data; and

generating a composite image data by aligning the images represented by the multi-viewpoint image data after the filter processing and combining the images represented by the multi-viewpoint image data after the alignment,

wherein a difference between the distance of the subject and the virtual focus distance increases when a difference in a pixel position of the subject between the images represented by the multi-viewpoint image data after the alignment increases.

16. A program stored in a non-transitory computer readable storage medium for causing a computer to perform the image processing method according to claim 15 .