Method for virtual image synthesis

1. A method for synthesizing virtual images from a sequence of texture images and a sequence of corresponding depth images, and wherein each depth image includes pixels, and each pixel stores a depth d at a pixel location I(x, y), comprising a processor for performing steps of the method, comprising the steps of:

preprocessing, each depth image, to produce a corresponding preprocessed depth image;

selecting, a first reference image and a second reference image from the sequence of texture images;

applying, to the preprocessed depth images corresponding to the reference images, depth-based 3D warping to produce a corresponding warped images;

applying, to the first and second reference images, depth-based histogram matching to produce a corresponding histogram image using the warped images, wherein the depth-based histogram matching reduces illumination differences between the two reference images by adjusting a distribution of pixel intensities;

applying, to each histogram image, base plus assistant image blending to produce a corresponding virtual image; and

applying, to each virtual image, depth-based in-painting to synthesize a virtual image, and further comprising:

constructing, for the first and second reference images, a cumulative reference histogram of pixel intensities;

determining a cumulative warped histogram of pixel intensities based on a linear combination of cumulative reference histograms corresponding to each reference image;

generating, for the first and second reference images, a mapping function based on the corresponding cumulative reference histograms and cumulative warped histogram; and

applying, for each pixel in the first and second reference images, the corresponding mapping function to produce the histogram images.

2. The method of claim 1 , wherein the depth preprocessing further comprises:

sliding a window across each pixel in the depth image, wherein a size of the window covers a set of pixels centered at the pixel;

assigning a single representative depth from the set of pixel in the window to the pixel to produce a processed depth image; and

filtering each pixel in the processed depth image to correct outlier depths without blurring depth discontinuities to produce the preprocessed depth image.

3. The method of claim 1 , wherein the preprocessing corrects errors and enhances spatial and temporal consistencies of depths at depth discontinuities.

4. The method of claim 1 , wherein the depth-based 3D warping projects pixels in the reference images to the warped image based on the preprocessed depth and camera parameters that describe a geometry of a scene.

5. The method of claim 1 wherein the base plus assistant image blending blends the histogram images using the preprocessed depth images.

6. The method of claim wherein the depth-based in-painting fills remaining holes using a depth-based in-painting.

7. The method of claim 6 , in which the depth-based in-painting fill up the holes using background pixels as determined by the depth image.

8. The method of claim 1 , wherein the sequence of texture images and a sequence of corresponding depth images, are reconstructed outputs of a decoder.

9. The method of claim 1 , further comprising:

predicting a texture image in the sequence of texture images with the virtual image.

10. The method of claim 9 , wherein the predicting is pail of an encoder, and further comprising:

computing a residual between the texture image and the virtual image; and

encoding the residual as part of a compressed bitstream.

11. The method of claim 9 , wherein the predicting is part of a decoder, and further comprising:

decoding a residual representing the difference between the texture image and the virtual image; and

adding the residual to the virtual image to produce a reconstructed texture image.