Method and appratus with neural rendering based on view augmentation
A method and apparatus for neural rendering based on view augmentation are provided. A method of training a neural scene representation (NSR) model includes: receiving original training images of a target scene, the original training images respectively corresponding to base views of the target scene; generating augmented images of the target scene by warping the original training images, the augmented images respectively corresponding to new views of the target scene; performing background-foreground segmentation on the original training images and the augmented images to generate segmentation masks; and training a neural scene representation (NSR) model to be configured for volume rendering of the target scene by using the original training images, the augmented images, and the segmentation masks.
1 . A method of training a neural scene representation (NSR) model, the method comprising:
receiving original training images of a target scene, the original training images respectively corresponding to base views of the target scene;
generating augmented images of the target scene by warping the original training images, the augmented images respectively corresponding to new views of the target scene, wherein the generating the augmented images comprises:
determining a transformation function for transforming a camera pose of a first of the base views of a first of the original training images into a camera pose of a first of the new views of a first of the augmented images; and
generating a first augmented image by warping a first original training image using an intrinsic camera parameter of the first original training image, an original depth map corresponding to the first original training image, and the transformation function;
performing background-foreground segmentation on the original training images and the augmented images to generate segmentation masks; and
training the NSR model to be configured for volume rendering of the target scene by using the original training images, the augmented images, and the segmentation masks,
wherein the training of the NSR model comprises:
performing primary training of the NSR model using the original training images, the augmented images, the segmentation masks, and a first loss function; and
performing secondary training of the NSR model using the original training images and a second loss function,
wherein the first loss function is based on a pixel error between (i) an actual pixel value from the original training images and the augmented images and (ii) a pixel value estimated by the NSR model, and
wherein the second loss function is based on a pixel error between the original training images and a synthesized image estimated by the NSR model, semantic consistency between the original training images and the synthesized image, and uncertainty of transmittance based on a ray.
2 . The method of claim 1 , wherein the performing of the primary training comprises:
selecting a first sample image from the original training images and the augmented images;
determining a first query output of the NSR model according to a first query input defining a first ray;
determining a target area to which the first ray belongs from among a foreground area of the first sample image and a background area of the first sample image, based on the segmentation masks; and
determining a loss value of the first loss function based on an actual pixel value of a first pixel of the target area specified by the first ray and an estimated pixel value according to the first query input.
3 . The method of claim 2 , wherein the determining of the target area comprises:
dividing the foreground area of the first sample image and the background area of the first sample image by applying a first of the segmentation masks corresponding to the first sample image to the first sample image;
when the first ray indicates the foreground area of the first sample image, determining the foreground area of the first sample image to be the target area; and
when the first ray indicates the background area of the first sample image, determining the background area of the first sample image to be the target area.
4 . The method of claim 1 , wherein the performing of the secondary training comprises:
generating a first synthesized image according to a first ray set of a first of the original training images by using the NSR model;
estimating first semantic characteristics of patches of the first original training image and second semantic characteristics of patches of the first synthesized image;
determining semantic consistency between the first original training image and the first synthesized image based on a difference between the first semantic characteristics and the second semantic characteristics; and
determining a loss value of the second loss function based on the determined semantic consistency.
5 . The method of claim 1 , wherein the performing of the secondary training comprises:
based on products of volume densities and transmittances of sample points of rays of the first original training image among the original training images, determining weights of the sample points; and
determining a loss value of the second loss function based on the weights of the rays.
6 . The method of claim 1 , wherein a number of original training images is limited to a predetermined number.
7 . An apparatus comprising:
one or more processors; and
a memory storing instructions configured to cause the one or more processors to:
receive original training images of a target scene, the original training images respectively corresponding to base views of the target scene,
generate augmented images of the target scene by warping the original training images, the augmented images respectively corresponding to new views of the target scene,
determine foreground-background segmentation masks of the original training images and the augmented images by performing foreground-background segmentation on the original training images and the augmented images, and
train a neural scene representation (NSR) model to be configured for volume rendering of the target scene by using the original training images, the augmented images, and the foreground-background segmentation masks,
wherein, to generate the augmented images, the instructions stored in the memory are further configured to cause the one or more processors to:
determine a transformation function for transforming a base camera pose of a first original training image of the original training images into a new camera pose of a first augmented image of the augmented images, and
generate the first augmented image by warping the first original training image using a camera intrinsic parameter of the first original training image, an original depth map corresponding to the first original training image, and the transformation function,
wherein, to train the NSR model, the instructions are further configured to cause the one or more processors to:
perform primary training of the NSR model using the original training images, the augmented images, the foreground-background segmentation masks, and a first loss function, and
perform secondary training of the NSR model using the original training images and a second loss function,
wherein the first loss function is based on a pixel error between an actual pixel value of the original training images and the augmented images and a pixel value estimated by the NSR model, and
wherein the second loss function is based on a pixel error between the original training images and a synthesized image estimated by the NSR model, semantic consistency between the original training images and the synthesized image, and uncertainty of transmittance based on a ray.
8 . The apparatus of claim 7 , wherein the original training images are respectively associated with base camera poses, the augmented images are respectively associated with new camera poses, and wherein the training of the NSR model also uses the base camera poses and the new camera poses.
9 . The apparatus of claim 7 , wherein, to perform the primary training, the instructions are further configured to cause the one or more processors to:
select a first sample image from the original training images and the augmented images, determine a first query output of the NSR model according to a first query input indicating a first ray,
determine a target area to which the first ray belongs among a foreground area of the first sample image and a background area of the first sample image, based on the foreground-background segmentation masks, and
determine a loss value of the first loss function based on an actual pixel value of a first pixel of the target area specified by the first ray and an estimated pixel value according to the first query output.
10 . The apparatus of claim 9 , wherein, to determine the target area, the instructions are further configured to cause the one or more processors to:
divide the foreground area of the first sample image and the background area of the first sample image by applying a first of the foreground-background segmentation masks corresponding to the first sample image to the first sample image,
when the first ray indicates the foreground area of the first sample image, determine the foreground area of the first sample image to be the target area, and
when the first ray indicates the background area of the first sample image, determine the background area of the first sample image to be the target area.
11 . The apparatus of claim 7 , wherein, to perform the secondary training, the instructions are further configured to cause the one or more processors to:
generate a first synthesized image according to a first ray set of a first of the original training images by using the NSR model,
estimate first semantic characteristics of multi-level patches of the first original training image and second semantic characteristics of multi-level patches of the first synthesized image, determine semantic consistency between the first original training image and the first synthesized image based on a difference between the first semantic characteristics and the second semantic characteristics, and
determine a loss value of the second loss function based on the determined semantic consistency.
12 . An electronic device comprising:
a camera generating original training images of respective original camera poses of a target scene, the original training images respectively corresponding to base views of the target scene; and
one or more processors;
a memory storing instructions configured to cause the one or more processors to:
generate augmented images of respective augmentation-image camera
poses for the target scene by warping the original training images, the augmented images respectively corresponding to new views of the target scene,
determine segmentation masks for dividing areas of the original training
images and the augmented images by performing segmentation on the original training images and the augmented images, and
train a neural scene representation (NSR) model used for volume rendering for the target scene by using the original training images and their respective original camera poses, the augmented images and their respective augmentation-image camera poses, and the segmentation masks,
wherein, to generate the augmented images, the instructions are further configured to cause the one or more processors to:
determine a transformation function for transform a first original camera pose of a first of the original training images into a first of the augmentation-image camera poses of a first of the augmented images, and
generate the first of the augmented images by warping a first original training image using a camera intrinsic parameter of the first of the original training images, an original depth map corresponding to the first of the original training images, and the transformation function,
wherein, to train the NSR model, the instructions are further configured to cause the one or more processors to:
perform primary training of the NSR model using the original training images, the augmented images, the segmentation masks, and a first loss function, and
perform secondary training of the NSR model using the original training images and a second loss function,
wherein the first loss function is based on a pixel error between an actual pixel value of the original training images and the augmented images and a pixel value estimated by the NSR model, and
wherein the second loss function is based on a pixel error between the original training images and a synthesized image estimated by the NSR model, semantic consistency between the original training images and the synthesized image, and uncertainty of transmittance based on a ray.