Volumetric video with auxiliary patches
Methods and devices for encoding and decoding data representative of a 3D scene are disclosed. A set of first patches is generated from a first MVD content acquired from a first region of the 3D scene. A patch is a part of one of the views of the MVD content. A set of second patches is generated from a second MVD content acquired from a second region of the 3D scene. An atlas packing first and second patches is generated and associated with metadata indicating, for a patch of the atlas, whether the patch is a first or a second patch At the decoding side, first patches are used for rendering the viewport image and second patches are used for pre-processing or post-processing the viewport image.
1. A method for encoding a 3D scene in a data stream, the method comprising:
generating a set of first patches from a first multi-view-plus-depth (MVD) content acquired for a rendering of the 3D scene, a first patch being a part of one of the views of the first MVD content;
generating a set of second patches from a second MVD content, wherein the second patches are not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene use, a second patch being a part of one of the views of the second MVD content;
generating a first atlas packing said first patches and a second atlas packing said second patches, an atlas being an image packing patches according to an atlas layout; and
encoding said first and second atlases in said data stream.
2. The method of claim 1 , wherein the second MVD content is acquired in a resolution lower than a resolution of the first MVD content.
3. The method of claim 1 , wherein a patch is a part of one view of a MVD content obtained by removing information redundancy between views of the MVD content.
4. The method of claim 1 , wherein said first and second patches are packed in a common atlas, the common atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.
5. The method of claim 1 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.
6. The method of claim 5 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.
7. A method for decoding a 3D scene from a data stream, the method comprising:
decoding said data stream to retrieve a first atlas and a second atlas, wherein the first atlas and the second atlas are images packing first or second patches respectively according to an atlas layout, a first patch being a part of one view of a first MVD content acquired for a rendering of the 3D scene, a second patch being a part of one view of a second MVD content, being not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene;
using second patches for a pre-processing and/or a post-processing of the rendering of the 3D scene; and
rendering a viewport image from a point of view within the 3D scene by using first patches.
8. The method of claim 7 , wherein the second MVD content has a resolution lower than a resolution of the first MVD content.
9. The method of claim 7 , wherein the first atlas and the second atlas are a unique atlas, the unique atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.
10. The method of claim 7 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.
11. The method of claim 10 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.
12. A device for encoding a 3D scene in a data stream, the device comprising a memory associated with a processor configured for:
generating a set of first patches from a first multi-view-plus-depth (MVD) content acquired for a rendering of the 3D scene, a first patch being a part of one of the views of the first MVD content;
generating a set of second patches from a second MVD content, wherein the second patches are not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene, a second patch being a part of one of the views of the second MVD content;
generating a first atlas packing said first patches and a second atlas packing said second patches, an atlas being an image packing patches according to an atlas layout; and
encoding said first and second atlases in said data stream.
13. The device of claim 12 , wherein the second MVD content is acquired in a resolution lower than a resolution of the first MVD content.
14. The device of claim 12 , wherein a patch is a part of one view of a MVD content obtained by removing information redundancy between views of the MVD content.
15. The device of claim 12 , wherein said first and second patches are packed in a unique atlas, the unique atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.
16. The device of claim 12 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.
17. The device of claim 16 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.
18. A device for decoding a 3D scene from a data stream, the device comprising a processor configured for:
decoding said data stream to retrieve a first atlas and a second atlas, wherein the first atlas and the second atlas are images packing first or second patches respectively according to an atlas layout, a first patch being a part of one view of a first MVD content acquired for a rendering of the 3D scene, a second patch being a part of one view of a second MVD content, being not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene use;
using second patches for a pre-processing and/or a post-processing of the rendering of the 3D scene; and
rendering a viewport image from a point of view within the 3D scene by using first patches.
19. The device of claim 18 , wherein the second MVD content has a resolution lower than a resolution of the first MVD content.
20. The device of claim 18 , wherein the first atlas and the second atlas are a unique atlas, the unique atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.
21. The device of claim 18 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.
22. The device of claim 21 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.