Method and apparatus for coding and decoding volumetric video with view-driven specularity
Methods and devices are provided for encoding, transmitting and decoding 3DoF+ volumetric video. At the encoding stage one input view (among all the input ones) is selected to convey the viewport dependent light effect and its id is transmitted to the decoder as an extra metadata. On the decoder side, when patches coming from this selected view are available for the rendering of the viewport, they are preferentially used regarding the other candidates whatever the view to synthesize position.
1. A method for encoding a 3D scene in a data stream, the 3D scene being provided as a Multi-view plus Depth frame, the method comprising:
selecting a view from the Multi-view plus Depth frame as a reference view based on a field of view coverage of the 3D scene from an average position in a viewing bounding box in the 3D scene;
generating an atlas image, the atlas image packing patches, a patch being an image part of a view of the Multi-view plus Depth frame, the atlas image comprising information representative of the 3D scene visible from the viewing bounding box;
generating metadata comprising information associating each patch with a view, and an identifier of the selected reference view; and
encoding the atlas and the metadata in the data stream.
2. The method of claim 1 , wherein the patches are obtained by pruning redundant information between views and conserving overlapping pixels between patches.
3. The method of claim 1 , wherein the reference view is packed in the atlas as an unpruned patch.
4. A non-transitory computer readable storage medium having stored thereon instructions for causing one or more processors to implement the method of claim 1 .
5. A method for rendering a viewport image representative of a 3D scene visible from a viewing bounding box in the 3D scene, the method comprising:
decoding an atlas image and associated metadata from a data stream, the atlas image packing patches, a patch being an image part of a view of a Multi-view plus Depth frame, the atlas image comprising information representative of the 3D scene visible from the viewing bounding box, and the metadata comprising information associating each patch with a view, and an identifier of a reference view, a patch associated with the reference view being a reference patch; and
when two patches contribute to the viewport image, rendering the viewport image for a current point of view within the viewing bounding box if one of the two patches is a reference patch, using only the contribution of the reference patch for the viewport image.
6. The method of claim 5 , wherein the reference view is one of the patches of the atlas.
7. A device for encoding a 3D scene in a data stream, the 3D scene being provided as a Multi-views plus Depth frame, the device comprising a processor configured for:
selecting a view from the Multi-view plus Depth frame as a reference view based on a field of view coverage of the 3D scene from an average position in a viewing bounding box in the 3D scene;
generating an atlas image, the atlas image packing patches, a patch being an image part of a view of the Multi-view plus Depth frame, the atlas image comprising information representative of the 3D scene visible from the viewing bounding box;
generating metadata comprising information associating each patch with a view, and an identifier of the selected reference view; and
encoding the atlas and the metadata in the data stream.
8. The device of claim 7 , wherein the patches are obtained by pruning redundant information between views and conserving overlapping pixels between patches.
9. The device of claim 7 , wherein the reference view is packed in the atlas as an unpruned patch.
10. A device for rendering a viewport image representative of a 3D scene visible from a viewing bounding box in the 3D scene, the device comprising a processor configured for:
decoding an atlas image and associated metadata from the data stream, the atlas image packing patches, a patch being an image part of a view of a Multi-view plus Depth frame, the atlas image comprising information representative of the 3D scene visible from the viewing bounding box, and the metadata comprising information associating each patch with a view, and an identifier of a reference view, a patch associated with the reference view being a reference patch; and
rendering the viewport image for a current point of view within the viewing bounding box by:
when two patches contribute to the viewport image, if one of the two patches is a reference patch, using only the contribution of the reference patch for the viewport image.
11. The device of claim 10 , wherein the reference view is one of the patches of the atlas.
12. A non-transitory computer readable storage medium having stored thereon instructions for causing one or more processors to implement the method of claim 5 .