Methods, devices and stream for encoding and decoding three degrees of freedom and volumetric compatible video stream
A sequence of three-dimension scenes is encoded as a video by an encoder and transmitted to a decoder which retrieves the sequence of 3D scenes. Points of a 3D scene visible from a determined point of view are encoded as a color image in a first track of the stream in order to be decodable independently from other tracks of the stream. The color image is compatible with a three degrees of freedom rendering. Depth information and depth and color of residual points of the scene are encoded in separate tracks of the stream and are decoded only in case the decoder is configured to decode the scene for a volumetric rendering.
1. A method of generating a stream representative of a 3D scene, the method comprising:
generating first color data by projecting first points of the 3D scene visible for 360 degrees from a first point of view onto a first color map, said first color data being a color image adapted to a rendering enabling three degrees of freedom of rotation around said first point of view;
generating first depth data by projecting said first points onto a first depth map, said first depth data being encoded as patch pictures packed in a depth atlas associated with metadata comprising information mapping a depth patch picture with a pixel area of said color image, wherein a patch picture comprises the points projected onto a projection map which define an area of adjacent pixels in the projection map and which are depth consistent, and wherein an atlas is an image gathering patch pictures;
generating second color data and second depth data representative of second points of the 3D scene visible for 360 degrees from a plurality of second points of view within a space of view around said first point of view, wherein said second points are visible from at least one of the plurality of second points of view and invisible from said first point of view; and
encoding, in said stream, a header, a first element of syntax encoding said first color data, and a second element of syntax encoding said first depth data, said second color data, said second depth data, and said metadata.
2. The method of claim 1 , wherein the 3D scene is obtained in the form of a Multiview plus depth (MVD) frame and wherein generating second color data and second depth data is performed by removing pixels of views of the MVD frame associated with points of the 3D scene visible from said first point of view.
3. The method of claim 1 , wherein generating second color data and second depth data is performed by projecting residual points of the 3D scene onto a second color map and onto a second depth map, said residual points being points of the 3D scene visible from within the space of view, and invisible from said first point of view.
4. The method of claim 3 , wherein said second color data are encoded as patch pictures packed in a color atlas and wherein said second depth data are encoded as patch pictures in a second depth atlas and associated with metadata comprising information mapping a depth patch picture with a color patch of said color atlas.
5. A device for encoding data representative of a 3D scene into a stream, the device comprising a processor configured to:
generate first color data by projecting first points of the 3D scene visible for 360 degrees from a first point of view onto a first color map according to a first projection mapping, said first color data being a color image adapted to a rendering enabling three degrees of freedom of rotation around said first point of view;
generate first depth data by projecting said first points onto a first depth map, said first depth data being encoded as patch pictures packed in a depth atlas associated with metadata comprising information mapping a depth patch picture with a pixel area of said color image, wherein a patch picture comprises the points projected onto a projection map which define an area of adjacent pixels in the projection map and which are depth consistent, and wherein an atlas is an image gathering patch pictures;
generate second color data and second depth data representative of second points of the 3D scene visible for 360 degrees from a plurality of second points of view within a space of view around said first point of view, wherein said second points are visible from at least one of the plurality of second points of view and invisible from said first point of view; and
encode, in said stream, a header, a first element of syntax encoding said first color data, and a second element of syntax encoding said first depth data, said second color data, said second depth data, and said metadata.
6. The device of claim 5 , wherein the 3D scene is obtained in the form of a Multiview plus depth (MVD) frame and wherein generating second color data and second depth data is performed by removing pixels of views of the MVD frame associated with points of the 3D scene visible from said first point of view.
7. The device of claim 5 , wherein generating second color data and second depth data is performed by projecting residual points of the 3D scene onto a second color map and onto a second depth map, said residual points being points of the 3D scene visible from the space of view, and invisible from said first point of view.
8. The device of claim 7 , wherein said second color data are encoded as patch pictures packed in a color atlas and wherein said second depth data are encoded as patch pictures in a second depth atlas and associated with metadata comprising information mapping a depth patch picture with a color patch of said color atlas.
9. A method of decoding a 3D scene implemented in a rendering device configured for a volumetric rendering, the method comprising:
obtaining a stream comprising a header, a first element of syntax, and a second element of syntax;
decoding from the first element of syntax, first color data being a color image encoded by projecting first points of the 3D scene visible for 360 degrees from a first point of view, said color image adapted to a rendering enabling three degrees of freedom of rotation around said first point of view; and
decoding from the second element of syntax, first depth data encoded as patch pictures packed in a first depth atlas obtained by projecting said first points, second color data, second depth data, and metadata comprising information mapping a depth patch picture with a pixel area of said color image, wherein the second depth data are representative of second points of the 3D scene visible for 360 degrees from a plurality of second points of view within a space of view around said first point of view, and wherein said second points are visible from at least one of the plurality of second points of view and invisible from said first point of view;
wherein a patch picture comprises the points projected onto a projection map which define an area of adjacent pixels in the projection map and which are depth consistent, and wherein an atlas is an image gathering patch pictures.
10. The method of claim 9 , further comprising:
decoding first depth data according to said metadata; and
decoding second color data and second depth data.
11. The method of claim 10 , wherein said second color data and second depth data are encoded as a MVD frame.
12. The method of claim 10 , wherein said second color data and second depth data are a projection of residual points of the 3D scene from said first point of view, said residual points being points of the 3D scene visible from said first point of view.
13. The method of claim 12 , wherein said second color data are encoded as patch pictures packed in a color atlas and wherein said second depth data are encoded as patch pictures in a second depth atlas associated with metadata comprising information mapping a depth patch picture to a color patch of said color atlas.
14. A rendering device configured for a volumetric rendering comprising a processor configured to:
obtain a stream comprising a header, a first element of syntax, and a second element of syntax;
decode from the first element of syntax, first color data being a color image encoded by projecting first points of the 3D scene visible for 360 degrees from a first point of view, said color image adapted to a rendering enabling three degrees of freedom of rotation around said first point of view; and
decode from the second element of syntax, first depth data encoded as patch pictures packed in a first depth atlas obtained by projecting said first points, second color data, second depth data, and metadata comprising information mapping a depth patch picture with a pixel area of said color image, wherein the second depth data are representative of second points of the 3D scene visible for 360 degrees from a plurality of second points of view within a space of view around said first point of view, and wherein said second points are visible from at least one of the second points of view and invisible from said first point of view;
wherein a patch picture comprises the points projected onto a projection map which define an area of adjacent pixels in the projection map and which are depth consistent, and wherein an atlas is an image gathering patch pictures.
15. The device of claim 14 , wherein the processor is further configured to:
decode first depth data according to said metadata; and
decode second color data and second depth data.
16. A non-transitory computer readable medium comprising computer-executable instructions to enable a processor to perform:
generating first color data by projecting first points of the 3D scene visible for 360 degrees from a first point of view onto a first color map, said first color data being a color image adapted to a rendering enabling three degrees of freedom of rotation around said first point of view;
generating first depth data by projecting said first points onto a first depth map, said first depth data being encoded as patch pictures packed in a depth atlas associated with metadata comprising information mapping a depth patch picture with a pixel area of said color image, wherein a patch picture comprises the points projected onto a projection map which define an area of adjacent pixels in the projection map and which are depth consistent, and wherein an atlas is an image gathering patch pictures;
generating second color data and second depth data representative of second points of the 3D scene visible for 360 degrees from a plurality of second points of view within a space of view around said first point of view, wherein said second points are visible from at least one of the plurality of second points of view and invisible from said first point of view; and
encoding, in said stream, a header, a first element of syntax encoding said first color data, and a second element of syntax encoding said first depth data, said second color data, said second depth data, and said metadata.
17. A non-transitory computer readable medium comprising computer-executable instructions to enable a rendering device configured for a volumetric rendering to perform:
obtaining a stream comprising a header, a first element of syntax, and a second element of syntax;
decoding from the first element of syntax, first color data being a color image encoded by projecting first points of the 3D scene visible for 360 degrees from a first point of view, said color image adapted to a rendering enabling three degrees of freedom of rotation around said first point of view; and
decoding from the second element of syntax, first depth data encoded as patch pictures packed in a first depth atlas obtained by projecting said first points, second color data, second depth data, and metadata comprising information mapping a depth patch picture with a pixel area of said color image, wherein the second depth data are representative of second points of the 3D scene visible for 360 degrees from a plurality of second points of view within a space of view around said first point of view, and wherein said second points are visible from at least one of the plurality of second points of view and invisible from said first point of view;
wherein a patch picture comprises the points projected onto a projection map which define an area of adjacent pixels in the projection map and which are depth consistent, and wherein an atlas is an image gathering patch pictures.