Image processing apparatus and image processing method
The present disclosure relates to an image processing apparatus and an image processing method that make it possible to generate a texture image of high picture quality at a predetermined viewpoint using an omnidirectional image. A plurality of real cameras image a plurality of reference images at a plurality of viewpoint positions in a plurality of sight line directions. A great weight is applied to a reference image in a sight line direction close to a virtual viewpoint specified in response to a movement of a head mounted display and having a virtual viewpoint position and a sight line direction to generate a virtual viewpoint image by weighted averaging, and the virtual viewpoint image is displayed on the head mounted display. The present disclosure can be applied, for example, to a home server or the like that generates a display image of a predetermined viewpoint from an omnidirectional image.
1. An image processing apparatus, comprising:
a virtual viewpoint image generation section configured to:
acquire metadata including information that specifies a plurality of reference images, wherein
the metadata further includes information for application of a weight to each reference image of the plurality of reference images based on similarities between a sight line direction of a determined virtual viewpoint and a plurality of sight line directions of the plurality of reference images;
apply the weight to each reference image of the plurality of reference images specified by the metadata; and
generate a virtual viewpoint image to be imaged at the determined virtual viewpoint having a virtual viewpoint position and the sight line direction, wherein
the virtual viewpoint image is generated by a weighted averaging process based on the applied weight to each reference image of the plurality of reference images, and
the plurality of reference images is imaged at a plurality of viewpoint positions in the plurality of sight line directions.
2. The image processing apparatus according to claim 1 , wherein the virtual viewpoint image generation section is further configured to mix the plurality of reference images based on the sight line direction of the determined virtual viewpoint and the plurality of sight line directions in which the plurality of reference images is imaged to generate the virtual viewpoint image.
3. The image processing apparatus according to claim 1 , wherein the virtual viewpoint image generation section is further configured to apply the weight individually to each reference image of the plurality of reference images such that a greater weight is set to a reference image having a higher similarity among the plurality of reference images.
4. The image processing apparatus according to claim 3 , wherein the similarities between the sight line direction of the determined virtual viewpoint and the plurality of sight line directions of the plurality of reference images are one of:
angles defined by the sight line direction of the determined virtual viewpoint and the plurality of sight line directions of the plurality of reference images, or
distances between the determined virtual viewpoint and the plurality of viewpoint positions of the plurality of reference images.
5. The image processing apparatus according to claim 1 , wherein
the weight is applied to each reference image of the plurality of reference images based on the similarities of the plurality of sight line directions of the plurality of reference images for a plurality of micro-parallax virtual viewpoints that provide different micro parallaxes with respect to the determined virtual viewpoint, such that a greater weight is set to a reference image having a higher similarity among the plurality of reference images, and
the plurality of reference images specified by the metadata includes reference images that have the greatest weight among the plurality of reference images.
6. The image processing apparatus according to claim 1 , wherein a reference image corresponding to the determined virtual viewpoint and the plurality of reference images of a plurality of micro-parallax virtual viewpoints at which micro parallaxes are generated with respect to the determined virtual viewpoint constitute a stereo image.
7. The image processing apparatus according to claim 1 , wherein
each reference image of the plurality of reference images is an omnidirectional image, and
the virtual viewpoint image generation section is further configured to generate, for each face of a plurality of faces that configure the omnidirectional image, the virtual viewpoint image to be imaged at the determined virtual viewpoint having the virtual viewpoint position and the sight line direction from the plurality of reference images.
8. The image processing apparatus according to claim 1 , wherein the plurality of reference images includes a first layer image having a first texture image and a first depth image of an omnidirectional image.
9. The image processing apparatus according to claim 8 , wherein the plurality of reference images further includes a second layer image having a second texture image and a second depth image in an occlusion region at a viewpoint of the first layer image.
10. The image processing apparatus according to claim 1 , further comprising a head mounted display configured to detect the determined virtual viewpoint and display the virtual viewpoint image.
11. An image processing method, comprising:
acquiring metadata including information that specifies a plurality of reference images, wherein
the metadata further includes information for application of a weight to each reference image of the plurality of reference images based on similarities between a sight line direction of a determined virtual viewpoint and a plurality of sight line directions of the plurality of reference images;
applying the weight to each reference image of the plurality of reference images specified by the metadata; and
generating a virtual viewpoint image to be imaged at the determined virtual viewpoint having a virtual viewpoint position and the sight line direction, wherein
the virtual viewpoint image is generated by a weighted averaging process based on the weight applied to each reference image of the plurality of reference images, and
the plurality of reference images is imaged at a plurality of viewpoint positions in the plurality of sight line directions.
12. An image processing apparatus, comprising:
a metadata generation section configured to:
apply a weight to each reference image of a plurality of reference images based on similarities between a sight line direction of a determined virtual viewpoint and a plurality of sight line directions of the plurality of reference images; and
generate metadata for generation of a virtual viewpoint image to be imaged at the determined virtual viewpoint having a virtual viewpoint position and the sight line direction, wherein
the metadata is generated based on the weight applied to each reference image of the plurality of reference images, and
the plurality of reference images is imaged at a plurality of viewpoint positions in the plurality of sight line directions.
13. The image processing apparatus according to claim 12 , wherein the metadata generation section is further configured to mix the plurality of reference images based on the determined virtual viewpoint and the plurality of sight line directions in which the plurality of reference images is imaged to generate the metadata.
14. The image processing apparatus according to claim 12 , wherein the metadata generation section is further configured to:
mix the plurality of reference images by a weighted averaging process;
generate the virtual viewpoint image; and
generate information that specifies a reference image that has the greatest weight among the plurality of reference images as the metadata.
15. The image processing apparatus according to claim 12 , wherein the metadata generation section is further configured to:
generate first metadata including first information, wherein the first information corresponds to:
weights set individually to each reference image of the plurality of reference images based on similarities between a micro-parallax virtual viewpoint having a micro parallax with respect to the determined virtual viewpoint and the plurality of sight line directions of the plurality of reference images,
generation of a first virtual viewpoint image, and
specification of a first reference image that has the greatest weight among the plurality of reference images; and
generate second metadata including second information, wherein the second information corresponds to:
the weights set individually to each reference image of the plurality of reference images based on the similarities between the determined virtual viewpoint and the plurality of sight line directions corresponding to the plurality of reference images,
generation of a second virtual viewpoint image, and
specification of a second reference image that has the greatest weight among the plurality of reference images.
16. The image processing apparatus according to claim 12 , wherein the plurality of reference images includes a first layer image having a first texture image and a first depth image of an omnidirectional image.
17. The image processing apparatus according to claim 16 , wherein the plurality of reference images further includes a second layer image having a second texture image and a second depth image in an occlusion region at a viewpoint of the first layer image.
18. An image processing method, comprising:
applying a weight to each reference image of a plurality of reference images based on similarities between a sight line direction of a determined virtual viewpoint and a plurality of sight line directions of the plurality of reference images; and
generating metadata for generating a virtual viewpoint image to be imaged at the determined virtual viewpoint having a virtual viewpoint position and the sight line direction, wherein
the metadata is generated based on the weight applied to each reference image of the plurality of reference images, and
the plurality of reference images is imaged at a plurality of viewpoint positions in the plurality of sight line directions.
19. An image processing apparatus, comprising:
a virtual viewpoint image generation section configured to:
apply a weight to each reference image of a plurality of reference images based on similarities between a sight line direction of a determined virtual viewpoint and a plurality of sight line directions of the plurality of reference images, wherein
each reference image of the plurality of reference images is an omnidirectional image; and
generate, for each face of a plurality of faces that configure the omnidirectional image, a virtual viewpoint image to be imaged at the determined virtual viewpoint having a virtual viewpoint position and the sight line direction from the plurality of reference images, wherein
the virtual viewpoint image is generated based on the applied weight to each reference image of the plurality of reference images, and
the plurality of reference images is imaged at a plurality of viewpoint positions in the plurality of sight line directions.