Multi-view coding with efficient residual handling
A gain in multi-view coding is achieved as follows: the residual signal involved with coding a dependent view of the multi-view signal is predicted from a reference residual signal of the current picture of the reference view using block-granular disparity-compensated prediction, i.e. using disparity compensated prediction with a disparity defined at, and varying with, block granularity so that each block of the current picture of the dependent view has its own disparity displacement such as its own disparity vector, associated therewith. In other words, a remaining similarity between the residual signal involved with predictively coding the reference view is used in order to predict the residual signal involved with predictively coding the dependent view.
1. A method for reconstructing a multi-view signal coded in a data stream, comprising:
obtaining a disparity vector with respect to a dependent coding block in a picture of a dependent view of a multi-view signal, the disparity vector representing a disparity between the dependent coding block and a reference-view coding block in a picture of a reference view of the multi-view signal;
identifying the reference-view block in the picture of the reference view based on the disparity vector;
obtaining, from the data stream, a reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between the reference-view coding block and a prediction of the reference-view coding block;
estimating a dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block;
adding a remaining signal corresponding to the dependent-view residual signal to the estimated dependent-view residual signal to obtain a reconstructed dependent-view residual signal; and
reconstructing the dependent coding block based on the reconstructed dependent-view residual signal and a prediction of the dependent coding block.
2. The method of claim 1 , wherein the obtaining the reference-view residual signal includes extracting the reference-view residual signal from the data stream.
3. The method of claim 1 , further comprising:
extracting, from the data stream, a second order prediction residual associated with the dependent coding block;
refining the estimated dependent-view residual signal for the dependent coding block based on the second order prediction residual to generate a refined estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the refined estimated dependent-view residual signal.
4. The method of claim 1 , further comprising:
extracting, from the data stream, disparity vector residual associated with the dependent coding block;
refining the disparity vector for the dependent coding block based on the disparity vector residual to generate a refined disparity vector; and
identifying the reference-view coding block based on the refined disparity vector.
5. The method of claim 1 , wherein
the reference-view residual signal is represented in a transform domain; and
the estimated dependent-view residual signal is determined in the transform domain.
6. The method of claim 1 , wherein the obtaining of the disparity vector includes estimating the disparity vector based on temporal prediction using information related to a reference block in a previously reconstructed picture of the dependent view.
7. An apparatus for reconstructing a multi-view signal coded in a data stream, comprising:
a disparity estimator configured for obtaining a disparity vector with respect to a dependent coding block in a picture of a dependent view of a multi-view signal, the disparity vector representing a disparity between the dependent coding block and a reference-view coding block in a picture of a reference view of the multi-view signal; and
a dependent-view reconstructor configured for, using a processor:
identifying the reference-view block in the picture of the reference view based on the disparity vector;
obtaining, from the data stream, a reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between the reference-view coding block and a prediction of the reference-view coding block;
estimating a dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block;
adding a remaining signal corresponding to the dependent-view residual signal to the estimated dependent-view residual signal to obtain a reconstructed dependent-view residual signal; and
reconstructing the dependent coding block based on the reconstructed dependent-view residual signal and a prediction of the dependent coding block.
8. The apparatus of claim 7 , further comprising a reference-view reconstructor configured for obtaining the reference-view residual signal by extracting the reference-view residual signal from the data stream.
9. The apparatus of claim 7 , wherein the dependent-view reconstructor is further configured for:
extracting, from the data stream, a second order prediction residual associated with the dependent coding block;
refining the estimated dependent-view residual signal for the dependent coding block based on the second order prediction residual to generate a refined estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the refined estimated dependent-view residual signal.
10. The apparatus of claim 7 , wherein the dependent-view reconstructor is further configured for:
extracting, from the data stream, disparity vector residual associated with the dependent coding block;
refining the disparity vector for the dependent coding block based on the disparity vector residual to generate a refined disparity vector; and
identifying the reference-view coding block based on the refined disparity vector.
11. The apparatus of claim 7 , wherein
the reference-view residual signal is represented in a transform domain; and
the estimated dependent-view residual signal is determined in the transform domain.
12. A machine readable non-transitory medium having data recorded thereon for reconstructing a multi-view signal coded in a data stream, the data, when read by the machine, cause the machine to perform:
obtaining a disparity vector with respect to a dependent coding block in a picture of a dependent view of a multi-view signal, the disparity vector representing a disparity between the dependent coding block and a reference-view coding block in a picture of a reference view of the multi-view signal;
identifying the reference-view block in the picture of the reference view based on the disparity vector;
obtaining, from the data stream, a reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between the reference-view coding block and a prediction of the reference-view coding block;
estimating a dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block;
adding a remaining signal corresponding to the dependent-view residual signal to the estimated dependent-view residual signal to obtain a reconstructed dependent-view residual signal; and
reconstructing the dependent coding block based on the reconstructed dependent-view residual signal and a prediction of the dependent coding block.
13. The medium of claim 12 , wherein the obtaining the reference-view residual signal includes extracting the reference-view residual signal from the data stream.
14. The medium of claim 12 , wherein the data, when read by the machine, cause the machine to further perform the following:
extracting, from the data stream, a second order prediction residual associated with the dependent coding block;
refining the estimated dependent-view residual signal for the dependent coding block based on the second order prediction residual to generate a refined estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the refined estimated dependent-view residual signal.
15. The medium of claim 12 , wherein the data, when read by the machine, cause the machine to further perform the following:
extracting, from the data stream, disparity vector residual associated with the dependent coding block;
refining the disparity vector for the dependent coding block based on the disparity vector residual to generate a refined disparity vector; and
identifying the reference-view coding block based on the refined disparity vector.
16. The medium of claim 12 , wherein
the reference-view residual signal is represented in a transform domain; and
the estimated dependent-view residual signal is determined in the transform domain.
17. A method for encoding a multi-view signal into a data stream, comprising:
determining a reference-view residual signal associated with a picture of a reference view of the multi-view signal, wherein the reference-view residual signal is to be used to obtain an estimate of a dependent-view residual signal for a dependent coding block in a picture of a dependent view of the multi-view signal via:
obtaining a disparity vector with respect to the dependent coding block in the picture in the dependent view, the disparity vector representing a disparity between the dependent coding block and the reference-view coding block,
identifying the reference-view block in the picture of the reference view based on the disparity vector,
obtaining the reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between a reference-view coding block and a prediction of the reference-view coding block,
estimating the dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the portion of the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block, and
determining a remaining signal that represents a difference between the dependent-view residual signal and the estimated dependent-view residual signal; and
inserting, into the data stream, the reference-view residual signal and the remaining signal.
18. The method of claim 17 , wherein the obtaining the reference-view residual signal includes extracting the reference-view residual signal from the data stream.
19. The method of claim 17 , the estimating the dependent-view residual signal includes:
extracting, from the data stream, a second order prediction residual associated with the dependent coding block;
refining the estimated dependent-view residual signal for the dependent coding block based on the second order prediction residual to generate a refined estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the refined estimated dependent-view residual signal.
20. The method of claim 17 , the estimating the dependent-view residual signal includes:
extracting, from the data stream, disparity vector residual associated with the dependent coding block;
refining the disparity vector for the dependent coding block based on the disparity vector residual to generate a refined disparity vector; and
identifying the reference-view coding block based on the refined disparity vector.
21. The method of claim 17 , wherein
the reference-view residual signal is represented in a transform domain; and
the estimated dependent-view residual signal is determined in the transform domain.
22. The method of claim 17 , wherein the obtaining of the disparity vector includes estimating the disparity vector based on temporal prediction using information related to a reference block in a previously reconstructed picture of the dependent view.
23. An apparatus for encoding a multi-view signal into a data stream, comprising:
a reference-view encoder configured for, using a processor,
determining a reference-view residual signal associated with a picture of a reference view of the multi-view signal, and
inserting, into the data stream, the reference-view residual signal and a remaining signal, wherein the reference-view residual signal is to be used to obtain an estimate of a dependent-view residual signal for a dependent coding block in a picture of a dependent view of the multi-view signal via:
obtaining a disparity vector with respect to the dependent coding block in the picture in the dependent view, the disparity vector representing a disparity between the dependent coding block and the reference-view coding block,
identifying the reference-view block in the picture of the reference view based on the disparity vector,
obtaining the reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between a reference-view coding block and a prediction of the reference-view coding block,
estimating the dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block, and
determining the remaining signal that represents a difference between the dependent-view residual signal and the estimated dependent-view residual signal.
24. The apparatus of claim 23 , the estimating the dependent-view residual signal includes:
extracting, from the data stream, a second order prediction residual associated with the dependent coding block;
refining the estimated dependent-view residual signal for the dependent coding block based on the second order prediction residual to generate a refined estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the refined estimated dependent-view residual signal.
25. The apparatus of claim 23 , the estimating the dependent-view residual signal includes:
extracting, from the data stream, disparity vector residual associated with the dependent coding block;
refining the disparity vector for the dependent coding block based on the disparity vector residual to generate a refined disparity vector; and
identifying the reference-view coding block based on the refined disparity vector.
26. The apparatus of claim 23 , wherein
the reference-view residual signal is represented in a transform domain; and
the estimated dependent-view residual signal is determined in the transform domain.
27. A machine readable non-transitory medium having data recorded thereon for encoding a multi-view signal into a data stream, wherein the data, when read by the machine, cause the machine to perform the following:
determining a reference-view residual signal associated with a picture of a reference view of the multi-view signal, wherein the reference-view residual signal is to be used to obtain an estimate of a dependent-view residual signal for a dependent coding block in a picture of a dependent view of the multi-view signal via:
obtaining a disparity vector with respect to the dependent coding block in the picture in the dependent view, the disparity vector representing a disparity between the dependent coding block and the reference-view coding block,
identifying the reference-view block in the picture of the reference view based on the disparity vector,
obtaining the reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between a reference-view coding block and a prediction of the reference-view coding block,
estimating the dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block, and
determining a remaining signal that represents a difference between the dependent-view residual signal and the estimated dependent-view residual signal; and
inserting, into the data stream, the reference-view residual signal and the remaining signal.
28. The medium of claim 27 , wherein the obtaining the reference-view residual signal includes extracting the reference-view residual signal from the data stream.
29. The medium of claim 27 , wherein the data, when read by the machine, cause the machine to further perform the following:
extracting, from the data stream, a second order prediction residual associated with the dependent coding block;
refining the estimated dependent-view residual signal for the dependent coding block based on the second order prediction residual to generate a refined estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the refined estimated dependent-view residual signal.
30. The medium of claim 27 , wherein the data, when read by the machine, cause the machine to further perform the following:
extracting, from the data stream, disparity vector residual associated with the dependent coding block;
refining the disparity vector for the dependent coding block based on the disparity vector residual to generate a refined disparity vector; and
identifying the reference-view coding block based on the refined disparity vector.
31. The medium of claim 27 , wherein
the reference-view residual signal is represented in a transform domain; and
the estimated dependent-view residual signal is determined in the transform domain.
32. A machine readable non-transitory medium for storing data associated with a multi-view signal, comprising:
a data stream stored in the non-transitory machine readable medium, the data stream comprising a remaining signal and a reference-view residual signal associated with a picture of a reference view of the multi-view signal, wherein the reference-view residual signal is to be used to obtain an estimate of a dependent-view residual signal for a dependent coding block in a picture of a dependent view of the multi-view signal via:
obtaining a disparity vector with respect to the dependent coding block in the picture in the dependent view, the disparity vector representing a disparity between the dependent coding block and the reference-view coding block,
identifying the reference-view block in the picture of the reference view based on the disparity vector,
obtaining the reference-view residual signal associated with the reference-view block, wherein the reference-view residual signal represents a difference between a reference-view coding block and a prediction of the reference-view coding block,
estimating the dependent-view residual signal for the dependent coding block in the picture of the dependent view based on the reference-view residual signal using the disparity vector, wherein the dependent-view residual signal represents a difference between the dependent coding block and a prediction of the dependent coding block, and
determining the remaining signal that represents a difference between the dependent-view residual signal and the estimated dependent-view residual signal; and
reconstructing the dependent coding block based on the reconstructed dependent-view residual signal and a prediction of the dependent coding block.
33. The machine readable non-transitory medium of claim 32 , wherein the obtaining of the disparity vector includes estimating the disparity vector based on temporal prediction using information related to a reference block in a previously reconstructed picture of the dependent view.