Methods and devices for encoding and decoding three degrees of freedom and volumetric compatible video stream
A sequence of three-dimension scenes is encoded as a video by an encoder and transmitted to a decoder which retrieves the sequence of 3D scenes. Points of a 3D scene visible from a determined point of view are encoded as a color image in a first track of the stream in order to be decodable independently from other tracks of the stream. The color image is compatible with a three degrees of freedom rendering. Depth information and depth and color of residual points of the scene are encoded in separate tracks of the stream and are decoded only in case the decoder is configured to decode the scene for a volumetric rendering.
1. A method of generating a stream representative of a 3D scene, the method comprising:
generating first color data by projecting points of the 3D scene visible from a point of view onto a first color map according to a first projection mapping, the first color data being a color image adapted to a rendering enabling 3 degrees of freedom of rotation around said point of view;
generating first depth data by projecting points of the 3D scene visible from said point of view onto a first depth map according to a second projection mapping; said first depth data being encoded as patch pictures packed in a first depth patch atlas associated with a first patch data item list, a first patch data item comprising information mapping a depth patch picture with a pixel area of said color image;
generating second color data and second depth data representative of points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view; and
encoding, in said stream, the first color data in a first element of syntax and said first depth data, said second color data and said second depth data in at least a second element of syntax.
2. The method of claim 1 , wherein the 3D scene is obtained in the form of a Multiview plus depth (MVD) frame and wherein generating second color data and second depth data is performed by removing pixels of views of the MVD frame associated with points of the 3D scene visible from said point of view.
3. The method of claim 1 , wherein generating second color data and second depth data is performed by projecting residual points of the 3D scene onto a second color map and onto a second depth map according to a third projection mapping, said residual points being points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view.
4. The method of claim 3 , wherein said second color data are encoded as patch pictures packed in a color patch atlas and wherein said second depth data are encoded as patch pictures in a second depth patch atlas and associated with a second patch data item list, a second patch data item comprising information mapping a depth patch picture with a color patch of said second patch atlas.
5. A device for encoding data representative of a 3D scene into a stream, the device comprising a memory associated with at least a processor configured to:
generate first color data by projecting points of the 3D scene visible from a point of view onto a first color map according to a first projection mapping, the first color data being a color image compatible for a rendering enabling 3 degrees of freedom of rotation around said point of view;
generate first depth data by projecting points of the 3D scene visible from said point of view onto a first depth map according to said second projection mapping; said first depth data being encoded as patch pictures packed in a first depth patch atlas associated with a first patch data item list, a first patch data item comprising information mapping a depth patch picture with a pixel area of said color image;
generate second color data and second depth data representative of points being points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view; and
encode, in said stream, the first color data in a first element of syntax and said first depth data, said second color data and said second depth data in at least a second element of syntax.
6. The device of claim 5 , wherein the 3D scene is obtained in the form of a Multiview plus depth (MVD) frame and wherein generating second color data and second depth data is performed by removing pixels of views of the MVD frame associated with points of the 3D scene visible from said point of view.
7. The device of claim 5 , wherein generating second color data and second depth data is performed by projecting residual points of the 3D scene onto a second color map and onto a second depth map according to a third projection mapping, said residual points being points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view.
8. The device of claim 7 , wherein said second color data are encoded as patch pictures packed in a color patch atlas and wherein said second depth data are encoded as patch pictures in a second depth patch atlas and associated with a second patch data item list, a second patch data item comprising information mapping a depth patch picture with a color patch of said second patch atlas.
9. A method of decoding a 3D scene, the method comprising:
obtaining a stream comprising:
a first element of syntax carrying first color data representative of the 3D scene, the first color data being a color image encoded by projecting points of the 3D scene visible from a point of view according to a first projection mapping, said color image being compatible for a rendering enabling 3 degrees of freedom of rotation around said point of view, and
at least a second element of syntax carrying first depth data encoded by projecting said points of the 3D scene from the point of view according to said first projection mapping and second color data and second depth data representative of points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view; wherein said first depth data are encoded as patch pictures packed in a first depth patch atlas associated with a first patch data item list, a first patch data item comprising information mapping a depth patch picture with a pixel area of said color image; and
decoding the color image from said first color data.
10. The method of claim 9 comprising:
decoding first depth data from the at least a second element of syntax of the stream, depth data being representative of a depth of points of the 3D scene visible from the point of view;
decoding second color data and second depth data from the at least a second element of syntax of the stream.
11. The method of claim 9 , wherein said second color data and second depth data are encoded as an MVD frame.
12. The method of claim 9 , wherein said second color data and second depth data are a projection of residual points of the 3D scene from said point of view according to a second projection mapping, said residual points being points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view.
13. The method of claim 12 , wherein said second color data are encoded as patch pictures packed in a color patch atlas and wherein said second depth data are encoded as patch pictures in a second depth patch atlas and associated with a second patch data item list, a second patch data item comprising information linking a depth patch picture to a color patch of said second patch atlas.
14. A device for decoding a 3D scene, the device comprising a memory associated with at least a processor configured to:
obtain a stream comprising:
a first element of syntax carrying first color data representative of the 3D scene, the first color data being a color image encoded by projecting points of the 3D scene visible from a point of view according to a first projection mapping, said color image being compatible for a rendering enabling 3 degrees of freedom of rotation around said point of view, and
at least a second element of syntax carrying first depth data encoded by projecting said points of the 3D scene from the point of view according to said first projection mapping and second color data and second depth data representative of points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view; wherein said first depth data are encoded as patch pictures packed in a first depth patch atlas associated with a first patch data item list, a first patch data item comprising information mapping a depth patch picture with a pixel area of said color image; and
decode the color image from said first color data.
15. The device of claim 14 wherein the processor is further configured to:
decode first depth data from the at least a second element of syntax of the stream, depth data being representative of a depth of points of the 3D scene visible from the point of view;
decode second color data and second depth data from the at least a second element of syntax of the stream.
16. The device of claim 14 , wherein said second color data and second depth data are encoded as an MVD frame.
17. The device of claim 14 , wherein said second color data and second depth data are a projection of residual points of the 3D scene from said point of view according to a second projection mapping, said residual points being points of the 3D scene invisible from said point of view and visible from viewpoints within a space of view around said first point of view.
18. The device of claim 17 wherein said second color data are encoded as patch pictures packed in a color patch atlas and wherein said second depth data are encoded as patch pictures in a second depth patch atlas and associated with a second patch data item list, a second patch data item comprising information linking a depth patch picture to a color patch of said second patch atlas.
19. The method of claim 9 , comprising transmitting the decoded 3D scene to a 3D scene renderer.
20. The device of claim 14 , comprising a transmitter configured to transmit the decoded 3D scene to a 3D scene renderer.