3D prediction method for video coding
Systems and methods are provided for using a Multi Focal Plane (MFP) prediction in predictive coding. The system detects a camera viewpoint change between a current frame from a current camera viewpoint to a previous frame from a previous camera viewpoint, decomposes a reconstructed previous frame to a plurality of focal planes, adjusts the plurality of focal planes from the previous camera viewpoint to correspond with the current camera viewpoint, generates an MFP prediction by summing pixel values of the adjusted plurality of focal planes along a plurality of optical axes from the current camera viewpoint, determines an MFP prediction error between the MFP prediction and the current frame, quantizes and codes the MFP prediction error, and transmits, to a receiver over a communication network, the camera viewpoint change and the coded quantized MFP prediction error for reconstruction of the current frame and display of the 3D scene.
1 . A method comprising:
detecting a camera viewpoint change between a current frame from a current camera viewpoint to a previous frame from a previous camera viewpoint, wherein the current frame represents 3D scene;
decomposing a reconstructed previous frame to a plurality of focal planes, wherein the reconstructed previous frame is based on the previous frame;
adjusting the plurality of focal planes from the previous camera viewpoint to correspond with the current camera viewpoint;
generating a Multi Focal Plane (MFP) prediction by summing pixel values of the adjusted plurality of focal planes along a plurality of optical axes from the current camera viewpoint;
determining an MFP prediction error between the MFP prediction and the current frame;
quantizing and coding the MFP prediction error; and
transmitting, to a receiver over a communication network, the camera viewpoint change and the coded quantized MFP prediction error for reconstruction of the current frame and display of the 3D scene.
2 . The method of claim 1 , further comprising:
generating a 2D intra prediction based on one or more reconstructed pixels of the current frame;
generating a 2D inter prediction based on one or more reconstructed previous frames;
determining a 2D intra prediction error between the 2D intra prediction and the current frame;
determining a 2D inter prediction error between the 2D inter prediction and the current frame;
determining a smallest error of the MFP prediction error, the 2D intra prediction error, and 2D inter prediction error;
selecting a mode corresponding to a type of prediction associated with the smallest error; and
transmitting the selected mode to the receiver over the communication network, wherein the selected mode corresponds to the MFP prediction in response to the MFP prediction error being the smallest error, and the transmitting the camera viewpoint change and the coded quantized MFP prediction error is in response to the MFP prediction error being the smallest error.
3 . The method of claim 1 , wherein detecting the camera viewpoint change comprises:
deriving the camera viewpoint change by using tracking information from position sensors.
4 . The method of claim 1 , wherein detecting the camera viewpoint change comprises:
deriving the camera viewpoint change by using the current frame and the previous frame.
5 . The method of claim 1 , wherein adjusting the plurality of focal planes comprises:
shifting each of the plurality of focal planes with a corresponding amount based on the camera viewpoint change; and
scaling each of the plurality of focal planes by a corresponding scale factor based on the camera viewpoint change, wherein a focal plane of the plurality of focal planes that is closer to the current camera viewpoint is shifted more and scaled larger in comparison to a focal plane of the plurality of focal planes that is further from the current camera viewpoint.
6 . The method of claim 1 , further comprising:
capturing the previous frame at a previous time;
coding the previous frame;
transmitting the coded previous frame to the receiver over the communication network; and
capturing the current frame at a current time being the previous time plus a frame delay.
7 . The method of claim 6 , wherein the frame delay is based on quantization accuracy in coding based on feedback from a status of the communication network or a status of the receiver over the communication network.
8 . The method of claim 1 , wherein the current frame and the previous frame are each represented using a video frame and a corresponding depth map.
9 . The method of claim 1 , wherein the plurality of optical axes intersect a focal plane of the adjusted plurality of focal planes at a plurality of intersection points, and a distance between a first intersection point and a second intersection point of the plurality of intersection points is less than a pixel spacing of an image corresponding to the focal plane.
10 . The method of claim 1 , wherein the plurality of focal planes are irregularly spaced in distance.
11 . A system comprising:
control circuitry configured to:
detect a camera viewpoint change between a current frame from a current camera viewpoint to a previous frame from a previous camera viewpoint, wherein the current frame represents 3D scene;
decompose a reconstructed previous frame to a plurality of focal planes, wherein the reconstructed previous frame is based on the previous frame;
adjust the plurality of focal planes from the previous camera viewpoint to correspond with the current camera viewpoint;
generate a Multi Focal Plane (MFP) prediction by summing pixel values of the adjusted plurality of focal planes along a plurality of optical axes from the current camera viewpoint;
determine an MFP prediction error between the MFP prediction and the current frame; and
quantize and code the MFP prediction error; and
input/output circuitry configured to:
transmit, to a receiver over a communication network, the camera viewpoint change and the coded quantized MFP prediction error for reconstruction of the current frame and display of the 3D scene.
12 . The system of claim 11 , wherein the control circuitry is further configured to:
generate a 2D intra prediction based on one or more reconstructed pixels of the current frame;
generate a 2D inter prediction based on one or more reconstructed previous frames;
determine a 2D intra prediction error between the 2D intra prediction and the current frame;
determine a 2D inter prediction error between the 2D inter prediction and the current frame;
determine a smallest error of the MFP prediction error, the 2D intra prediction error, and 2D inter prediction error; and
select a prediction mode corresponding to a type of prediction associated with the smallest error; and
wherein the input/output circuitry is further configured to:
transmit the selected mode to the receiver over the communication network, wherein the selected mode corresponds to the MFP prediction in response to the MFP prediction error being the smallest error, and the input/output control circuitry is configured to transmit the camera viewpoint change and the coded quantized MFP prediction error in response to the MFP prediction error being the smallest error.
13 . The system of claim 11 , wherein the control circuitry is configured to detect the camera viewpoint change by:
deriving the camera viewpoint change by using tracking information from position sensors.
14 . The system of claim 11 , wherein the control circuitry is configured to detect the camera viewpoint change by:
deriving the camera viewpoint change by using the current frame and the previous frame.
15 . The system of claim 11 , wherein the control circuitry is configured to adjust the plurality of focal planes by:
shifting each of the plurality of focal planes with a corresponding amount based on the camera viewpoint change; and
scaling each of the plurality of focal planes by a corresponding scale factor based on the camera viewpoint change, wherein a focal plane of the plurality of focal planes that is closer to the current camera viewpoint is shifted more and scaled larger in comparison to a focal plane of the plurality of focal planes that is further from the current camera viewpoint.
16 . The system of claim 11 , wherein the control circuitry is further configured to:
capture the previous frame at a previous time; and
code the previous frame;
wherein the input/output circuitry is further configured to:
transmit the coded previous frame to the receiver over the communication network; and
wherein the control circuitry is further configured to:
capture the current frame at a current time being the previous time plus a frame delay.
17 . The system of claim 16 , wherein the frame delay is based on quantization accuracy in coding based on feedback from a status of the communication network or a status of the receiver over the communication network.
18 . The system of claim 11 , wherein the current frame and the previous frame are each represented using a video frame and a corresponding depth map.
19 . The system of claim 11 , wherein the plurality of optical axes intersect a focal plane of the adjusted plurality of focal planes at a plurality of intersection points, and a distance between a first intersection point and a second intersection point of the plurality of intersection points is less than a pixel spacing of an image corresponding to the focal plane.
20 . The system of claim 11 , wherein the plurality of focal planes are irregularly spaced in distance.