High-resolution multiview-consistent rendering and alpha matting from sparse views
A method including capturing a first plurality of images that include a foreground object and a background, capturing a second plurality of images that include the background, generating an alpha matte based on the first plurality of images and the second plurality of images using a trained machine learned model trained using a loss function configured to cause the trained machine learned model to learn high-frequency details of the foreground object, generating a foreground object image based on the first plurality of images and the second plurality of images using the trained machine learned model, and synthesizing an image including the foreground object image and a second background scene using the alpha matte.
1 . A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, cause a computing system to:
generate an alpha matte based on a first plurality of images that include a foreground object on a first background scene and a second plurality of images including the first background scene using a model trained using a loss function configured to cause the model to determine a geometry of the foreground object that has three dimensions and first high-frequency details of the foreground object in a first stage and to determine second high-frequency details of the foreground object in a second stage;
generate a foreground object image based on the first plurality of images and the second plurality of images using the model; and
generate an image including the foreground object image and a second background scene using the alpha matte.
2 . The non-transitory computer-readable storage medium of claim 1 , wherein the loss function configured to cause the model to determine at least one of the first high-frequency details or the second high-frequency details of the foreground object includes a Sobel loss.
3 . The non-transitory computer-readable storage medium of claim 1 , wherein the loss function includes an annealing patch-wise smoothness loss.
4 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions further cause the computing system to:
capture the first plurality of images that include the foreground object and the first background scene using a sparse-view camera rig; and
capture the second plurality of images that include the second background scene using the sparse-view camera rig.
5 . The non-transitory computer-readable storage medium of claim 1 , wherein the model is a neural network including a Sobel field multilayer perceptron (MLP).
6 . The non-transitory computer-readable storage medium of claim 5 , wherein a Sobel map is used to supervise the training of the model.
7 . A method comprising:
selecting an alpha matte generated using a model;
selecting a first image that includes a foreground object generated using the model;
selecting a background image; and
generating a second image that includes the foreground object and the background image using the alpha matte, the first image and the alpha matte being generated using the model trained using a loss function configured to cause the model to determine a geometry of the foreground object that is three dimensional and first high-frequency details of the foreground object in a first stage and to determine second high-frequency details of the foreground object in a second stage.
8 . The method of claim 7 , wherein the model is trained using a first plurality of images including the foreground object and a background scene, a second plurality of images including the background scene, and the loss function includes a Sobel loss.
9 . The method of claim 8 , wherein the first plurality of images and the second plurality of images are captured using a camera rig having greater than two and less than twenty cameras.
10 . The method of claim 7 , wherein the loss function includes an annealing patch-wise smoothness loss.
11 . The method of claim 7 , wherein the model is a neural network including a Sobel field multilayer perceptron (MLP).
12 . The method of claim 11 , wherein a Sobel map is used to supervise training of the model.
13 . The method of claim 7 , wherein
the foreground object is generated with a new viewpoint,
the alpha matte is modified based on the foreground object with the new viewpoint, and
the generating of the second image includes removing a portion of the background image using the modified alpha matte and replacing the removed portion with the foreground object with the new viewpoint.
14 . A method comprising:
generating an alpha matte based on a first plurality of images that include a foreground object on a first background scene and a second plurality of images including the first background scene using a model trained using a loss function configured to cause the model to determine a geometry of the foreground object that has three dimensions and first high-frequency details of the foreground object in a first stage and to determine second high-frequency details of the foreground object in a second stage;
generating a foreground object image based on the first plurality of images and the second plurality of images using the model; and
generating an image including the foreground object image and a second background scene using the alpha matte.
15 . The method of claim 14 , wherein the loss function configured to cause model to determine at least one of the first high-frequency details or the second high-frequency details of the foreground object includes a Sobel loss.
16 . The method of claim 14 , wherein the loss function includes an annealing patch-wise smoothness loss.
17 . The method of claim 14 , wherein the method further comprises:
capturing the first plurality of images that include the foreground object and the first background scene using a sparse-view camera rig; and
capturing the second plurality of images that include the second background scene using the sparse-view camera rig.
18 . The method of claim 14 , wherein the model is a neural network including a Sobel field multilayer perceptron (MLP).
19 . The method of claim 18 , wherein a Sobel map is used to supervise training of the model.
20 . The method of claim 14 , wherein the generating of the image includes removing a portion of the second background scene using the alpha matte and replacing the removed portion with the foreground object image.