Different-view image generating apparatus and different-view image generating method

1. A different-viewpoint image generating apparatus comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by the processor, cause the different-viewpoint image generating apparatus to function as:

a different-viewpoint image generating unit configured to generate, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing;

a hole density calculating unit configured to calculate, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels;

a combination ratio calculating unit configured to calculate, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and

a different-viewpoint image combining unit configured to combine the two or more different-viewpoint images, based on the combination ratios of the processing units,

wherein the hole density calculating unit is configured to calculate, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and

wherein the hole density calculating unit is configured to calculate a plurality of the hole densities by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.

2. The different-viewpoint image generating apparatus according to claim 1 ,

wherein the executable instructions, when executed by the processor, cause the different-viewpoint image generating apparatus to further function as a hole area interpolating unit configured to interpolate each of the hole areas in the two or more different-viewpoint images, using pixel values within a corresponding one of the two or more different-viewpoint images,

wherein the different-viewpoint image combining unit is configured to combine the two or more different-viewpoint images including the interpolated hole areas, based on the combination ratios.

3. The different-viewpoint image generating apparatus according to claim 1 ,

wherein the executable instructions, when executed by the processor, cause the different-viewpoint image generating apparatus to further function as a hole area interpolating unit configured to interpolate the hole areas within images combined by the different-viewpoint image combining unit, using pixel values within the images.

4. The different-viewpoint image generating apparatus according to claim 1 ,

wherein the combination ratio calculating unit is configured to calculate, for the co-located processing units in the two or more different-viewpoint images, a plurality of the combination ratios which become larger as a plurality of the hole densities of the co-located processing units become smaller.

5. The different-viewpoint image generating apparatus according to claim 1 ,

wherein, when hole densities of co-located processing units are equal to each other between the two or more different-viewpoint images, the combination ratio calculating unit is configured to calculate, for the co-located processing units having the equal hole densities, combination ratios which become larger as the positions of the two or more different-viewpoint images become closer to the position of the virtual viewpoint position.

6. The different-viewpoint image generating apparatus according to claim 1 ,

wherein the processing unit is a pixel,

the hole density calculating unit is configured to calculate a hole density for each of pixels co-located in the two or more different-viewpoint images, and

the combination ratio calculating unit is configured to calculate, for each of the co-located pixels, the combination ratio of one of the two or more different-viewpoint images.

7. A different-viewpoint image generating method comprising:

generating, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing;

calculating, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels;

calculating, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and

combining the two or more different-viewpoint images, based on the combination ratios of the processing units,

wherein the calculating of the hole density comprises calculating, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and

wherein, in the calculating of the hole density, a plurality of the hole densities are calculated by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.

8. A non-transitory computer-readable recording medium for use in a computer, the recording medium having a computer program recorded thereon for causing the computer to execute the different-viewpoint image generating method according to claim 7 .

9. An integrated circuit comprising:

a different-viewpoint image generating circuit configured to generate, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing;

a hole density calculating circuit configured to calculate, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels;

a combination ratio calculating circuit configured to calculate, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and

a different-viewpoint image combining circuit configured to combine the two or more different-viewpoint images, based on the combination ratios of the processing units,

wherein the hole density calculating circuit is configured to calculate, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and

wherein the hole density calculating circuit is configured to calculate a plurality of the hole densities by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.

10. The different-viewpoint image generating apparatus according to claim 1 ,

wherein the hole density calculating unit is configured to calculate the plurality of the hole densities by assigning a larger weight to the hole area located at the central part of the window than the hole area located at the peripheral part of the window.