Multiview multiscale methods and apparatus for view synthesis
The present disclosure relates to method and apparatus for the estimation of an accurate multiplane image from a set of posed images. The accuracy of an MPI representation is measured by the performance of the views generated by it through inverse homography sampling and alpha compositing. A variable number of posed images at a possibly different resolution representative of scenes with arbitrary geometry are used as input of the system. A first level of three modules, possibly trained based on deep learning methods, is built for extracting view and scene features and color scores. Numerous levels of the same three modules may be built and organized in a structure where a level deals with down-scaled version of the posed images.
1. A method comprising:
operating a level of three convolutional network (CNN) modules:
a first CNN module being operated to extract view features from a set of posed images;
a second CNN module being operated to extract scene features of the set of posed images by using a result of the operating of the first CNN module; and
a third CNN module being operated to extract color scores of the set of posed images by using the result of the operating of the first CNN module and a result of the operating of the second CNN module; and
generating a multiplane image from the set of posed images, wherein a multiplane image is a stack of images comprising a color component and an alpha component, comprising, for each image of the stack:
the alpha component determined as a function of the result of the operating of the second CNN module; and
the color component determined by normalizing the result of the operating of the third CNN module.
2. The method of claim 1 , further comprising operating a second level of the three CNN modules, wherein:
the posed images in input of the CNN modules of the second level are downscaled by a factor; and
the results operated by the CNN modules of the second level are upscaled by the factor and are used in input of corresponding CNN modules of the first level.
3. A device comprising a processor configured for:
operating a level of three CNN modules:
a first CNN module being operated by the processor to extract view features from a set of posed images;
a second CNN module being operated by the processor to extract scene features of the set of posed images by using a result of operating of the first CNN module; and
a third CNN module being operated by the processor to extract color scores of the set of posed images by using the result of the operating of the first CNN module and a result of operating of the second CNN module; and
generating a multiplane image from the set of posed images, wherein a multiplane image is a stack of images comprising a color component and an alpha component, comprising, for each image of the stack:
the alpha component determined as a function of the result of the operating of the second CNN module; and
the color component determined by normalizing the result of the operating of the third CNN module.
4. The device of claim 3 , wherein the processor is further configured for operating a second level of the three CNN modules, wherein:
the posed images in input of the CNN modules of the second level are downscaled by a factor; and
the results operated by the CNN modules of the second level are upscaled by the factor and are used in input of corresponding CNN modules of the first level.
5. A method of training a system, the method comprising:
operating a level of three CNN modules:
a first CNN module being operated to extract view features from a set of first posed images;
a second CNN module being operated to extract scene features of the set of first posed images by using a result of the operating of the first CNN module; and
a third CNN module being operated to extract color scores of the set of first posed images by using the result of the operating of the first CNN module and a result of the operating of the second CNN module;
generating a multiplane image from the set of posed images, wherein a multiplane image is a stack of images comprising a color component and an alpha component, comprising, for each image of the stack:
the alpha component determined as a function of the result of the operating of the second CNN module; and
the color component determined by normalizing the result of the operating of the third CNN module;
generating a second posed image for a viewpoint from the multiplane image; and
back-propagating a gradient descent in the three CNN modules by comparing the second posed image with a third posed image captured at the viewpoint.
6. The method of claim 5 , further comprising operating a second level of the three CNN modules, wherein:
the posed images in input of the CNN modules of the second level are downscaled by a factor; and
the results of the operating of the CNN modules of the second level are upscaled by the factor and are used in input of corresponding CNN modules of the first level.
7. A device for training a system, the device comprising a processor configured for:
operating a level of three CNN modules:
a first CNN module being operated to extract view features from a set of first posed images;
a second CNN module being operated to extract scene features of the set of first posed images by using a result of the operating of the first CNN module; and
a third CNN module being operated to extract color scores of the set of first posed images by using the result of the operating of the first CNN module and a result of the operating of the second CNN module;
generating a multiplane image from the set of posed images, wherein a multiplane image is a stack of images comprising a color component and an alpha component, comprising, for each image of the stack:
the alpha component determined as a function of the result of the operating of the second CNN module; and
the color component determined by normalizing the result of the operating of the third CNN module;
generating a second posed image for a viewpoint from the multiplane image; and
back-propagating a gradient descent in the three CNN modules by comparing the second posed image with a third posed image captured at the viewpoint.
8. The device of claim 5 , wherein the processor is further configured for operating a second level of the three CNN modules, wherein:
the posed images in input of the CNN modules of the second level are downscaled by a factor; and
the results of the operating of the CNN modules of the second level are upscaled by the factor and are used in input of corresponding CNN modules of the first level.