Multi-view coding with effective handling of renderable portions
A proposed intermediate way of handling the renderable portion of the first view results in more efficient coding. Instead of omitting the coding of the renderable portion completely, even more efficient coding of multi-view signals entails merely suppressing the coding of the residual signal within the renderable portion, whereas the prediction parameter coding still takes place from the non-renderable portion of the multi-view signal across the renderable portion so that prediction parameters for the renderable portion may be exploited for predicting parameters for the non-renderable portion. The additional coding rate for transmitting the prediction parameters for the renderable portion may be kept low as this merely aims at forming a continuation of the parameter history across the renderable portion to serve as a basis for prediction parameters of other portions of the multi-view signal.
1. A decoder for reconstructing a multi-view signal from a data stream, comprising:
a data stream extractor configured to extract, using a processor from the data stream, a rendering flag associated with a renderable portion of a first view of the multi-view signal, the rendering flag being indicative of whether the renderable portion is to be replaced with a rendered portion generated, by view synthesis, based on a second view of the multi-view signal; and
a reconstructor configured to reconstruct, using the processor, at least a portion of the first view based on a residual signal, wherein the decoder is configured to decode in a first coding mode or a second coding mode depending on the rendering flag, wherein the decoder is configured to, in the first coding mode, render, by view synthesis, the renderable portion of the first view from the second view to generate the rendered portion and replace the renderable portion of a reconstructed first view by the rendered portion, with not performing the replacement in the second coding mode.
2. The decoder of claim 1 , wherein the data stream extractor is configured to extract, using the processor, a prediction parameter from the data stream, the decoder further comprising:
a view predictor configured to perform, using the processor, a block-based prediction of a block of a non-renderable portion of the first view based on the prediction parameter and a reference signal derived from a previously-reconstructed portion of the multi-view signal to acquire a prediction signal,
wherein the reconstructor configured to reconstruct the at least a portion of the first view based on the prediction signal and the residual signal.
3. The decoder of claim 2 , wherein the view predictor is configured to, in performing the block-based prediction, predict the block within the non-renderable portion using, as the prediction parameter, a motion vector or a disparity vector.
4. The decoder of claim 2 , wherein the view predictor is configured to perform the block-based prediction of a block of the renderable portion of the first view using another prediction parameter, wherein the block of the renderable portion and the block of the non-renderable portion are within different pictures of the first view.
5. The decoder of claim 1 , further comprising a determinator configured to determine the renderable portion of the first view using a depth map of a scene to which the first and second views belong.
6. An encoder for encoding a multi-view signal into a data stream, comprising:
a determinator configured to determine, using a processor, a renderable portion of a first view of the multi-view signal which is renderable, by view synthesis, based on a second view of the multi-view signal;
obtaining, using the processor, a rendering flag associated with the renderable portion of the first view; and
a data stream generator configured to encode, using the processor, the rendering flag and a residual signal into the data stream,
wherein, based on a first coding mode associated with the rendering flag, the renderable portion of the first view is rendered, by view synthesis, from the second view to generate the rendered portion and replace the renderable portion of a reconstructed first view by the rendered portion, while based on a second coding mode, the replacement of the renderable portion is not performed.
7. The encoder of claim 6 , further comprising a view predictor configured to perform, using the processor, a block-based prediction of a block of a non-renderable portion of the first view based on a prediction parameter and a previously-encoded portion of the multi-view signal to acquire a prediction signal.
8. The encoder of claim 7 , wherein the view predictor is configured to, in performing the block-based prediction, predict the block within the non-renderable portion using, as the prediction parameter, a motion vector or a disparity vector.
9. The encoder of claim 7 , wherein the view predictor is configured to perform the block-based prediction of a block of the renderable portion of the first view using another prediction parameter, wherein the block of the renderable portion and the block of the non-renderable portion are within different pictures of the first view.
10. The encoder of claim 7 , further comprising:
a reconstructor configured to reconstruct the first view, including the renderable portion, from the prediction signal to acquire a reference signal, wherein the view predictor is configured to perform the block-based prediction based on the reference signal.
11. The encoder of claim 7 , wherein the determinator is configured to render the renderable portion to acquire the rendered signal, and the encoder further comprises:
a reconstructor configured to reconstruct the non-renderable portion based on the residual signal and the prediction signal to acquire a reconstructed signal, wherein the reconstructed signal and the rendered signal form a reference signal, which is used by the view predictor to perform the block-based prediction.
12. The encoder of claim 6 , wherein the data stream generator is configured to encode position information indicating a position of the renderable portion into the data stream.
13. The encoder of claim 6 , wherein the determinator configured to determine the renderable portion of the first view using a depth map of a scene to which the first and second views belong.
14. A method for decoding a multi-view signal from a data stream, the method comprising:
extracting, using a processor from the data stream, a rendering flag associated with a renderable portion of a first view of the multi-view signal, the rendering flag being indicative of whether the renderable portion is to be replaced with a rendered portion generated, by view synthesis, based on a second view of the multi-view signal; and
reconstructing, using the processor, at least a portion of the first view based on a residual signal, wherein the decoder is configured to decode in a first coding mode or a second coding mode depending on the rendering flag, wherein the decoder is configured to, in the first coding mode, render, by view synthesis, the renderable portion of the first view from the second view to generate the rendered portion and replace the renderable portion of a reconstructed first view by the rendered portion, with not performing the replacement in the second coding mode.
15. The method of claim 14 , further comprising:
extracting, using the processor, a prediction parameter from the data stream;
performing, using the processor, a block-based prediction of a block of a non-renderable portion of the first view based on the prediction parameter and a reference signal derived from a previously-reconstructed portion of the multi-view signal to acquire a prediction signal; and
reconstructing the at least a portion of the first view based on the prediction signal and the residual signal.
16. The method of claim 15 , wherein performing the block-based prediction comprises predicting the block within the non-renderable portion using, as the prediction parameter, a motion vector or a disparity vector.
17. The method of claim 15 , further comprising performing the block-based prediction of a block of the renderable portion of the first view using another prediction parameter, wherein the block of the renderable portion and the block of the non-renderable portion are within different pictures of the first view.
18. The method of claim 14 , further comprising:
determining the renderable portion of the first view using a depth map of a scene to which the first and second views belong.
19. A method for storing a data stream, comprising storing, on a digital storage medium, a data stream from which a multi-view signal is decoded by a method according to claim 14 .