Layer-merging and post-warp error correction in multi-plane images
View Patent ↗A method for merging layers of multi-plane image (MPI) includes steps (1) to (5). Step (1) includes determining a layer-weight array by, for each of D1 image-layers each located at a respective one of D1 layer-depths, determining a respective total weight as an element-wise sum of a weighting-factor array of the image-layer. Step (2) includes interpolating the layer-weight array to yield an interpolated layer-weight signal. Step (3) includes partitioning the interpolated layer-weight signal into D2<D1 segments to yield D2 optimized layer-depth intervals. Step (4) includes determining, for each optimized layer-depth interval. (i) intra-interval layer-depths, of the D1 layer-depths, located within the optimized layer-depth interval; and (ii) corresponding intra-interval layer weights, of the D1 total weights. Step (5) includes determining, for each of the D2 optimized layer-depth intervals, a respective output layer-depth as an average of the intra-interval layer-depths each weighted by a respective intra-interval layer weight.
1 . A method for merging layers of a multi-plane image (MPI), comprising:
determining a layer-weight array {{a (s) }} by, for each image-layer of D1 image-layers of the MPI each located at a respective one of D1 layer-depths, determining a respective one of D1 total weights as an element-wise sum of a weighting-factor array {{W p (x, y)}} of the image-layer;
interpolating the layer-weight array to yield an interpolated layer-weight signal {{q (s) (z)}};
partitioning the interpolated layer-weight signal into D2 segments to yield D2 optimized layer-depth intervals {{with edges t p′ }}, D2 being less than D1;
determining, for each optimized layer-depth interval, (i) a number of intra-interval layer-depths, of the D1 layer-depths, located within the optimized layer-depth interval; and (ii) a number of intra-interval layer weights, of the D1 total weights, each associated with a respective one of the number of intra-interval layer-depths; and
determining, for each of the D2 optimized layer-depth intervals, a respective output layer-depth {{d′}} as an average of the number of intra-interval layer-depths each weighted by a respective one of the number of intra-interval layer weights.
2 . The method of claim 1 , further comprising, before determining the layer-weight array:
for each image-layer of the D1 image-layers, determining the weighting-factor array {{W p (x, y)}} of the image layer from (i) an alpha channel of the image-layer, and (ii) alpha channels of image-layers, of the D1 image-layers, having a layer-depth smaller than a layer-depth of the layer.
3 . The method of claim 2 , determining the weighting-factor array comprising: determining, for each pixel coordinate of the layer, a respective pixel-wise weighting factor from (i) the value of the alpha channel at the pixel coordinate, and (ii) values of the alpha channels at the pixel coordinate.
4 . The method of claim 1 , partitioning comprising minimizing an objective function that includes a monotonically non-decreasing function of the interpolated layer-weight signal.
5 . The method of claim 1 , each pair of adjacent layer-depths of the D1 layer-depths being separated by a same initial distance, and partitioning comprising:
determining D2 weighted depths {{r p′ }} by, for each of D2 layer-depth intervals each having a respective one of a plurality of lower-boundaries {{t p′ }}, determining a respective one of D2 weighted depths as an average of layer-depth values within the layer-depth interval weighted by values of the interpolated layer-weight signal within the layer-depth interval,
updating the D2 layer-depth intervals by, for each layer-depth interval of the D2 layer-depth intervals for which the initial distance is less than a difference between (i) an average R of the layer-depth interval's associated weighted depth and the subsequent weighted depth and (ii) the lower-boundary {{t p′ }} of the layer-depth interval, changing the value of the lower boundary to equal the average R; and
obtaining the D2 optimized layer-depth intervals by iterating said steps of determining D2 weighted depths and updating the D2 layer-depth intervals until the plurality of lower-boundaries converge to within a predetermined tolerance.
6 . The method of claim 5 , further comprising, before any instances of determining, initializing the D2 layer-depth intervals such that each of the D2 layer-depth intervals has an equal length.
7 . The method of claim 1 , each of the number of intra-interval layer-depths being a layer-depth of a respective one of a number of intra-interval image layers, of the D1 image layers, each having a respective one of a number of intra-interval texture channels
{
{
C
p
(
s
)
between pMin ana pMax}}, and further comprising:
determining, for each of the D2 optimized layer-depth intervals, a respective texture channel
{
{
C
p
′
′
(
s
)
}
}
as an average of the number of intra-interval texture channels weighted by a respective one of the number of intra-interval layer weights.
8 . The method of claim 1 , each of the number of intra-interval layer-depths being a layer-depth of a respective one of a number of intra-interval image layers, of the D1 image layers, each having a respective one of a number of intra-interval alpha channels {{A p between pMin and pMax}}, and further comprising:
determining, for each of the D2 optimized layer-depth intervals, a respective alpha channel
{
{
A
p
′
′
(
s
)
}
}
as an average of the number of intra-interval alpha-channels weighted by a respective one of the number of intra-interval layer weights.
9 . A method for merging layers of N multi-plane images (MPIs) from N different camera poses of a scene, comprising:
determining N interpolated layer-weight signals {{q (s) (z)}} by, for each of the N MPIs:
determining a layer-weight array {a (s) }} by, for each image-layer of D1 image-layers of the MPI each located at a respective one of D1 layer-depths, determining a respective one of D1 total weights as an element-wise sum of a weighting-factor array {{W p (x,y)}} of the image-layer; and
interpolating the layer-weight array to yield an interpolated layer-weight signal {{q (s) (z)}};
obtaining a global interpolated layer-weight signal {{q(z)}} as a sum of each of the N interpolated layer-weight signals;
partitioning the global interpolated layer-weight signal into D2 segments to yield D2 globally-optimized layer-depth intervals {{with edges t p′ }}, D2 being less than D1;
determining, for each of the D2 globally-optimized layer-depth intervals, (i) a number of intra-interval layer-depths, of the D1 layer-depths, located within the optimized layer-depth interval; and (ii) a number of intra-interval layer weights, of the D1 total weights, each associated with a respective one of the number of intra-interval layer-depths; and
determining for each of the D2 globally-optimized layer-depth intervals, a respective output layer-depth {{d′}} as an average of the number of intra-interval layer-depths weighted by a respective one of the number of intra-interval layer weights.
10 . The method of claim 9 , further comprising, for each of the N MPIs, determining a respective texture channel for each of the D2 optimized layer-depth intervals as an average of the number of intra-interval texture channels weighted by a respective one of the number of intra-interval layer weights.
11 . The method of claim 9 , further comprising, for each of the N MPIs, determining a respective alpha channel for each of the D2 optimized layer-depth intervals as an average of the number of intra-interval alpha-channels weighted by a respective one of the number of intra-interval layer weights.
12 . An image-layer merger comprising:
a processor; and
a memory storing a multi-plane image (MPI) and machine readable instructions that, when executed by a processor, cause the processor to execute the method of claim 1 .