Systems and methods for generating and coding multiple focal planes from texture and depth
System and methods are provided for using multi-format representation and coding of visual information. The system accesses an image data that comprises a texture data and a depth map; decomposes the depth map into a plurality of component depth maps (CDMs); and generates multiple focal planes (MFPs) comprising a plurality of focal planes. Each respective focal plane is based on the texture data and a respective CDM of the plurality of CDMs. The system selects a data subset including one or more of: (a) the texture data, (b) the depth map, (c) the plurality of CDMs, or (d) the plurality of focal planes; generates encoded data based on the selected data subset; and transmits, over a communication network, the encoded data to a client device to cause the client device to: generate for display or for further processing an image based on the encoded data.
1 . A method comprising:
accessing image data that comprises texture data and a depth map;
decomposing the depth map into a plurality of component depth maps (CDMs);
generating, from the texture data and the CDMs, a plurality of multiple focal plane (MFP) data structures, wherein each MFP data structure is produced by pixel-wise combining the texture data with a respective CDM;
generating, from the depth map and the CDMs, a plurality of CDM-weighted depth plane (CDMFP) data structures;
selecting, based on a selection criterion, a data subset from among: (a) the texture data, (b) the depth map, (c) the plurality of CDMs, (d) the plurality of MFP data structures, or (e) the plurality of CDMFP data structures;
generating encoded data based on the selected data subset to produce encoded data; and
transmitting, over a communication network, the encoded data to a client device to cause the client device to generate for display an image based on the encoded data.
2 . The method of claim 1 , wherein:
the plurality of CDMs is a set of n CDMs; and
the data subset comprises (a) the texture data and (b) n−1 selected CDMs.
3 . The method of claim 1 , wherein:
the plurality of MFP data structures is a set of n focal planes; and
the data subset comprises (a) the texture data and (b) n−1 selected focal planes.
4 . The method of claim 1 , wherein the data subset comprises (a) the depth map and (b) the plurality of MFP data structures.
5 . The method of claim 1 , wherein the data subset comprises (a) the plurality of CDMs and (b) the plurality of MFP data structures.
6 . The method of claim 1 , wherein:
the plurality of CDMs is a set of n CDMs;
the plurality of focal planes is a set of n focal planes, wherein each respective CDM of the plurality of CDMs and each respective focal plane of the plurality of MFP data structures corresponds to a respective depth range of the depth map; and
the data subset comprises (a) the texture data, (b) selected CDMs, and (c) selected focal planes, wherein a total number of (b) the selected CDMs and (c) the selected focal planes is n−1, and wherein each one of the selected CDMs and the selected focal planes corresponds to a different depth range of the depth map.
7 . The method of claim 6 , wherein the transmitting the encoded data to the client device causes the client device to generate for display the image by:
decoding the encoded data to generate (a) decoded texture data (b) decoded selected CDMs and (c) decoded selected focal planes, wherein a total number of (b) the decoded selected CDMs and (c) the decoded selected focal planes is n−1; and
generating, based at least in part on the decoded texture data, the decoded selected CDMs, and the decoded selected focal planes, n reformed CDMs and n reformed focal planes, wherein the n reformed CDMs and the n reformed focal planes contain omitted CDMs and omitted focal planes that were not included in the selected data subset.
8 . The method of claim 7 , wherein the generating the n reformed CDMs and the n reformed focal planes comprises:
generating a reformed CDM of the n reformed CDMs by:
summing n−1 reformed CDMs of the n reformed CDMs to generate a partial CDM sum; and
subtracting the partial CDM sum from a saturated matrix to generate the reformed CDM.
9 . The method of claim 7 , wherein the generating the n reformed CDMs and the n reformed focal planes comprises:
generating a reformed focal plane of the n reformed focal planes by:
summing n−1 reformed focal planes of the n reformed focal planes to generate a partial MFP sum; and
subtracting the partial MFP sum from the decoded texture data to generate the reformed focal plane.
10 . The method of claim 7 , wherein the generating the n reformed CDMs and the n reformed focal planes comprises:
generating, from a decoded CDM of the n reformed CDMs, a corresponding reformed focal plane of the n reformed focal planes by:
multiplying the decoded texture data by the decoded CDM to generate the corresponding reformed focal plane of the n reformed focal planes.
11 . The method of claim 7 , wherein the transmitting the encoded data to the client device causes the client device to generate for display the image by further:
generating a reformed depth map by inverse depth blending the n reformed CDMs.
12 . The method of claim 1 , wherein:
the plurality of CDMs is a set of n CDMs;
the plurality of focal planes is a set of n focal planes, wherein each respective CDM of the plurality of CDMs and each respective focal plane of the plurality of focal planes corresponds to a respective depth range of the depth map; and
the data subset comprises (a) the texture data, (b) selected CDMs, and (c) selected focal planes, wherein a total number of (b) the selected CDMs and (c) the selected focal planes is n, and wherein each one of the selected CDMs and the selected focal planes corresponds to a different depth range of the depth map.
13 . A system comprising:
control circuitry configured to:
access image data that comprises texture data and a depth map;
decompose the depth map into a plurality of component depth maps (CDMs);
generate, from the texture data and the CDMs, a plurality of multiple focal planes (MFP) data structures, wherein each MFP data structure is produced by pixel-wise combining the texture data with a respective CDM;
generate, from the depth map and the CDMs, a plurality of CDM-weighted depth plane (CDMFP) data structures;
select, based on a selection criterion, a data subset from among: (a) the texture data, (b) the depth map, (c) the plurality of CDMs, (d) the plurality of MFP data structures, or (e) the plurality of CDMFP data structures; and
generate encoded data based on the selected data subset to produce encoded data; and
input/output circuitry configured to:
transmit, over a communication network, the encoded data to a client device to cause the client device to generate for display an image based on the encoded data.
14 . The system of claim 13 , wherein:
the plurality of CDMs is a set of n CDMs;
the plurality of focal planes is a set of n focal planes, wherein each respective CDM of the plurality of CDMs and each respective focal plane of the plurality of focal planes corresponds to a respective depth range of the depth map; and
the data subset comprises (a) the texture data, (b) selected CDMs, and (c) selected focal planes, wherein a total number of (b) the selected CDMs and (c) the selected focal planes is n−1, and wherein each one of the selected CDMs and the selected focal planes corresponds to a different depth range of the depth map.
15 . The system of claim 14 , wherein the control circuitry configured to transmit the encoded data to the client device causes the client device to generate for display the image by:
decoding the encoded data to generate (a) decoded texture data (b) decoded selected CDMs and (c) decoded selected focal planes, wherein a total number of (b) the decoded selected CDMs and (c) the decoded selected focal planes is n−1; and
generating, based at least in part on the decoded texture data, the decoded selected CDMs, and the decoded selected focal planes, n reformed CDMs and n reformed focal planes, wherein the n reformed CDMs and the n reformed focal planes contain omitted CDMs and omitted focal planes that were not included in the selected data subset.
16 . The system of claim 15 , wherein the generating the n reformed CDMs and the n reformed focal planes comprises:
generating a reformed CDM of the n reformed CDMs by:
summing n−1 reformed CDMs of the n reformed CDMs to generate a partial CDM sum; and
subtracting the partial CDM sum from a saturated matrix to generate the reformed CDM.
17 . The system of claim 15 , wherein the generating the n reformed CDMs and the n reformed focal planes comprises:
generating a reformed focal plane of the n reformed focal planes by:
summing n−1 reformed focal planes of the n reformed focal planes to generate a partial MFP sum; and
subtracting the partial MFP sum from the decoded texture data to generate the reformed focal plane.
18 . The system of claim 15 , wherein the generating the n reformed CDMs and the n reformed focal planes comprises:
generating, from a decoded CDM of the n reformed CDMs, a corresponding reformed focal plane of the n reformed focal planes by:
multiplying the decoded texture data by the decoded CDM to generate the corresponding reformed focal plane of the n reformed focal planes.
19 . The system of claim 15 , wherein the control circuitry configured to transmit the encoded data to the client device causes the client device to generate for display the image by further:
generating a reformed depth map by inverse depth blending the n reformed CDMs.
20 . The system of claim 13 , wherein:
the plurality of CDMs is a set of n CDMs;
the plurality of focal planes is a set of n focal planes, wherein each respective CDM of the plurality of CDMs and each respective focal plane of the plurality of focal planes corresponds to a respective depth range of the depth map; and
the data subset comprises (a) the texture data, (b) selected CDMs, and (c) selected focal planes, wherein a total number of (b) the selected CDMs and (c) the selected focal planes is n, and wherein each one of the selected CDMs and the selected focal planes corresponds to a different depth range of the depth map.