Apparatus and method for generating new viewpoint image using few-shot 2D images
The present disclosure relates to an apparatus and a method for generating an image, which may generate a new viewpoint image using few-shot 2D images and is characterized by extracting and learning albedo representing intrinsic color information from the few-show 2D input images, intrinsically decomposing a color value synthesized in the form of patch from a new viewpoint using the few-shot 2D input images and extracting patch-wise albedo during learning, perform geometric alignment necessary for pixel correspondence between a new viewpoint image and a selected 2D input image using a depth value synthesized in the form of patch from the new viewpoint, and generating the new viewpoint image by calculating an albedo consistency loss between the patch-wise albedo.
1 . An apparatus for generating a new viewpoint image using few-shot 2D images, comprising:
a full-image intrinsic decomposition unit configured to receive the few-shot 2D images, intrinsically decompose each input image, and extract albedo representing intrinsic color information from each 2D input image;
a patch image intrinsic decomposition unit configured to learn using the albedo provided from the full-image intrinsic decomposition unit, and learn to minimize an albedo consistency loss, intrinsically decompose a color value synthesized in the form of patch from a new viewpoint using the few-shot 2D input images during learning, and extract patch-wise albedo;
an alignment unit configured to perform geometric alignment necessary for pixel correspondence between a new viewpoint image and a 2D input image selected from the few-shot 2D input images using a depth value synthesized in the form of patch from the new viewpoint, calculate an albedo consistency loss between the albedo extracted by the full-image intrinsic decomposition unit and the patch-wise albedo extracted by the patch image intrinsic decomposition unit based on the pixel correspondence, and provide the albedo consistency loss to the patch image intrinsic decomposition unit; and
an image generation unit configured to generate the new viewpoint image from the few-shot 2D input images by reflecting the patch-wise albedo extracted by the patch image intrinsic decomposition unit finishing learning.
2 . The apparatus of claim 1 , wherein the full-image intrinsic decomposition unit intrinsically decomposes each of the 2D input images offline using a given global context.
3 . The apparatus of claim 1 , wherein the alignment unit performs geometric alignment by performing projective transformation based on the depth value synthesized in the form of patch from the new viewpoint.
4 . The apparatus of claim 1 , wherein the alignment unit minimizes inaccurate pixel correspondence by reflecting a depth consistency loss of the selected 2D input image.
5 . The apparatus of claim 1 , wherein the albedo consistency loss is calculated by reflecting a weight term that minimizes a projection error.
6 . A method of generating a new viewpoint image using few-shot 2D images, comprising:
a full-image intrinsic decomposing operation of receiving the few-shot 2D images, intrinsically decomposing each input image, and extracting albedo representing intrinsic color information from each 2D input image;
a patch image intrinsic decomposing operation of learning using the albedo extracted in the full-image intrinsic decomposing operation, and learning to minimize an albedo consistency loss, intrinsically decomposing a color value synthesized in the form of patch from a new viewpoint using the few-shot 2D input images during learning, and extracting patch-wise albedo;
an aligning operation of performing geometric alignment necessary for pixel correspondence between a new viewpoint image and a 2D input image selected from the few-shot 2D input images using a depth value synthesized in the form of patch from the new viewpoint, calculating an albedo consistency loss between the albedo extracted in the full-image intrinsic decomposing operation and the patch-wise albedo extracted in the patch image intrinsic decomposing operation based on the pixel correspondence, and provide the albedo consistency loss to the patch image intrinsic decomposing operation; and
an image generating operation of generating the new viewpoint image from the few-shot 2D input images by intrinsically decomposing the color value synthesized in the form of patch from the new viewpoint using the few-shot 2D input images and reflecting the extracted patch-wise albedo, after the learning is finished.
7 . The method of claim 6 , wherein in the full-image intrinsic decomposing operation, each of the 2D input images is intrinsically decomposed offline using a given global context.
8 . The method of claim 6 , wherein in the aligning operation, geometric alignment is performed by performing projective transformation based on the depth value synthesized in the form of patch from the new viewpoint.
9 . The method of claim 6 , wherein in the aligning operation, inaccurate pixel correspondence is minimized by reflecting a depth consistency loss of the selected 2D input image.
10 . The method of claim 6 , wherein the albedo consistency loss is calculated by reflecting a weight term that minimizes a projection error.