Multi-view coding with exploitation of renderable portions
Rendered portions of a renderable portion of a first view of a multi-view signal are introduced into a prediction loop of a multi-view encoder to form a reference signal for the block-based prediction of the encoder's view predictor may improve the prediction and thereby increasing the coding efficiency. The introduction may be performed by completely inserting the renderable portion into the prediction loop to form new reference pictures with their own reference picture indices in addition to reference pictures obtained by the reconstructed version of the multi-view signal of the block-based prediction. Alternatively, the rendered portion may be introduced into the prediction loop completely, but with replacing the normal prediction reference signal, i.e., the normally reconstructed signal as obtained by block-based prediction. Alternatively, it may be signaled within the data stream which portions of the renderable portion are used to replace respective portions in the normally reconstructed signal.
1. A decoder for decoding a multi-view signal representing a video, comprising:
a renderer configured for rendering, using a processor, a renderable portion of a first view of the multi-view signal, by view synthesis, based on a second view of the multi-view signal to generate a rendered portion of the first view;
an extractor configured for extracting, from the data stream, using the processor, a first residual signal and a second residual signal;
a view predictor configured for performing, using the processor, a block-based motion prediction based on the rendered portion and a non-renderable portion of the first view, at least by
obtaining a first set of motion prediction parameters associated with the rendered portion, and
generating a prediction signal based on the rendered portion, the first set of motion prediction parameters, the non-renderable portion, and a second set of motion prediction parameters associated with the non-renderable portion;
a parameter predictor configured for obtaining, using the processor, the second set of motion prediction parameters based on the first set of motion prediction parameters and the second residual; and
a reconstructor configured for reconstructing the first view based on the prediction signal and the first residual.
2. The decoder of claim 1 , wherein the extractor is further configured for extracting, from the data stream, first position information indicative of the rendered portion of the first view.
3. The decoder of claim 1 , wherein the extractor is further configured for extracting, from the data stream, second position information indicative of the renderable portion.
4. The decoder of claim 1 , wherein the renderer is configured for rendering the renderable portion to generate the rendered portion based on depth/disparity map of a scene associated with the first and second views.
5. The decoder of claim 1 , wherein the reconstructor is configured for reconstructing, based on the prediction signal and the first residual signal, a reconstructed reference signal to be used by the view predictor for block-based prediction.
6. The decoder of claim 1 , wherein the view predictor is configured for performing block-based disparity compensated prediction to obtain a reference signal for a renderable portion of a third view.
7. The decoder of claim 1 , wherein the view predictor is configured to, in performing the block-based prediction, predict at least one first block within the renderable portion using first disparity vector data, and at least one second block within the non-renderable portion using second disparity vector data, wherein the parameter predictor is configured to predict the second disparity vector data from the first disparity vector data.
8. The decoder of claim 1 , wherein the non-renderable portion of the first view is disjoint to the renderable portion of the first view.
9. An encoder for encoding a multi-view signal representing a video into a data stream, comprising:
a determinator configured for rendering, using a processor, a renderable portion of a first view of the multi-view signal, by view synthesis based on a second view of the multi-view signal to generate a rendered portion of the first view;
a view predictor configured for performing, using the processor, a block-based motion prediction based on the rendered portion and a non-renderable portion of the first view, at least by
obtaining a first set of motion prediction parameters associated with the rendered portion,
obtaining a second set of motion prediction parameters associated with the non-renderable portion of the first view, and
generating a prediction signal based on the rendered portion, the first set of motion prediction parameters, the non-renderable portion, and the second set of motion prediction parameters;
a parameter predictor configured for predicting, using the processor, the second set of motion prediction parameters based on the first set of motion prediction parameters;
a residual generator configured for
determining, using the processor, a first residual signal based on the prediction signal and the first view of the multi-view signal, and
determining, using the processor, a second residual signal based on the first and second sets of motion prediction parameters; and
a data stream generator configured for inserting, using the processor, the first and second residual signals into the data stream.
10. The encoder of claim 9 , wherein the data stream generator is further configured for inserting first position information indicative of the rendered portion of the first view into the data stream.
11. The encoder of claim 9 , wherein the data stream generator is further configured for inserting second position information indicative of the renderable portion into the data stream.
12. The encoder of claim 9 , wherein the determinator is configured for rendering the renderable portion to generate the rendered portion based on depth/disparity map of a scene associated with the first and second views.
13. The encoder of claim 9 , further comprising a reconstructor configured for reconstructing, based on the prediction signal and the first residual signal, a reconstructed reference signal to be used by the view predictor for block-based prediction.
14. The encoder of claim 9 , wherein the view predictor is configured for performing block-based disparity compensated prediction to obtain a reference signal for a renderable portion of a third view.
15. A non-transitory computer readable medium having data recorded thereon for encoding a multi-view signal representing a video into a data stream, wherein the data, when read by the computer, cause the computer to perform:
rendering a renderable portion of a first view of the multi-view signal, by view synthesis based on a second view of the multi-view signal to generate a rendered portion of the first view;
performing a block-based motion prediction based on the rendered portion and a non-renderable portion of the first view, at least by obtaining a first set of motion prediction parameters associated with the rendered portion,
obtaining a second set of motion prediction parameters associated with the non-renderable portion of the first view, and
generating a prediction signal based on the rendered portion, the first set of motion prediction parameters, the non-renderable portion, and the second set of motion prediction parameters;
predicting the second set of motion prediction parameters based on the first set of motion prediction parameters;
determining a first residual signal based on the prediction signal and the first view of the multi-view signal;
determining a second residual signal based on the first and second sets of motion prediction parameters; and
inserting the first and second residual signals into the data stream.
16. The computer readable medium of claim 15 , when the data is read by the computer, further causing the computer to perform inserting first position information indicative of the rendered portion of the first view into the data stream.
17. The computer readable medium of claim 15 , when the data is read by the computer, further causing the computer to perform inserting second position information indicative of the renderable portion into the data stream.
18. The computer readable medium of claim 15 , wherein the step of rendering the renderable portion is based on depth/disparity map of a scene associated with the first and second views.
19. The computer readable medium of claim 15 , when the data is read by the computer, further causing the computer to perform reconstructing, based on the prediction signal and the first residual signal, a reconstructed reference signal to be used for block-based prediction.
20. The computer readable medium of claim 15 , wherein the non-renderable portion of the first view is disjoint to the renderable portion of the first view.