Methods and apparatuses for encoding, decoding and rendering 6DOF content from 3DOF+ composed elements
A volumetric content is encoded as a set of clusters by an encoder and transmitted to a decoder which retrieves the volumetric content. Clusters common to different viewpoints are obtained and mutualized. Clusters are projected onto 2D images and encoded as independent video streams. Reduction in visual artefacts and reduction of data for storage and streaming are achieved.
1 . A method for encoding a 3D scene, the method comprising:
clustering points of the 3D scene into a plurality of clusters according to a depth range of the points of the 3D scene, wherein the plurality of clusters comprises at least a background cluster and a foreground cluster;
obtaining a first set of 2D images by projecting clusters visible from a first set of points of view according to first projection parameters, wherein the first set of points of view is encompassed in a first viewing box defined in the 3D scene and comprises at least two points of view;
obtaining a second set of 2D images by projecting clusters visible from a second set of points of view according to second projection parameters, wherein the second set of points of view is encompassed in a second viewing box defined in the 3D scene different from the first viewing box;
identifying one or more common clusters visible from both the first set of points of view and the second set of points of view; and
encoding the first set of 2D images and the first projection parameters in a first data stream and each 2D image of the second set of 2D images and the second projection parameters in a set of distinct data streams, wherein 2D common images corresponding to the one or more common clusters are encoded only once.
2 . The method of claim 1 , wherein the clustering is further based on semantics associated with points of the 3D scene, on color of the points of the 3D scene, or on motion of points of the 3D scene.
3 . The method of claim 1 , further comprising:
encoding metadata comprising:
a list of viewing boxes defined in the 3D scene and a list of common clusters for the 3D scene; and
for each viewing box, a list of sets of clusters representative of the 3D scene for the respective viewing box and, for each set of clusters associated with the viewing box, identifiers of the common clusters, and the list of clusters other than the common clusters.
4 . A non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .
5 . A method for decoding a 3D scene, the method comprising:
decoding a first set of first 2D images from a first data stream and a second 2D image from each data stream of a set of distinct data streams, the first 2D images being representative of a projection according to first projection parameters of at least one cluster of points of the 3D scene visible from a first set of points of view encompassed in a first viewing box defined in the 3D scene, the second 2D images being representative of a projection according to second projection parameters of at least one cluster of points of the 3D scene visible from a second set of points of view encompassed in a second viewing box defined in the 3D scene different from the first viewing box, wherein 2D common images corresponding to common clusters to the first and second viewing boxes are decoded only once, and the points of the 3D scene being clustered into a plurality of clusters according to a depth range of the points of the 3D scene, wherein the plurality of clusters comprises at least a background cluster and a foreground cluster; and
un-projecting pixels of the first 2D images according to the first projection parameters and to the first set of points of view and un-projecting pixels of the second 2D images according to the second projection parameters and to the second set of points of view.
6 . The method of claim 5 , further comprising
obtaining metadata comprising:
a list of viewing boxes defined in the 3D scene and a list of common clusters for the 3D scene; and
for each viewing box, a list of sets of clusters representative of the 3D scene for the respective viewing box and, for each set of clusters associated with the viewing box, identifiers of the common clusters, and the list of clusters other than the common clusters; and
decoding 2D images from data streams comprising clusters of 3D points visible from a current point of view.
7 . A non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 5 .
8 . A device for encoding a 3D scene comprising a memory associated with a processor configured for:
clustering points of the 3D scene into a plurality of clusters according to a depth range of the points of the 3D scene, wherein the plurality of clusters comprises at least a background cluster and a foreground cluster;
obtaining a first set of 2D images by projecting clusters visible from a first set of points of view according to first projection parameters, wherein the first set of points of view is encompassed in a first viewing box defined in the 3D scene and comprises at least two points of view;
obtaining a second set of 2D images by projecting clusters visible from a second set of points of view according to second projection parameters, wherein the second set of points of view is encompassed in a second viewing box defined in the 3D scene different from the first viewing box;
identifying one or more common clusters visible from both the first set of points of view and the second set of points of view; and
encoding the first set of 2D images and the first projection parameters in a first data stream and each 2D image of the second set of 2D images and the second projection parameters in a set of distinct data streams, wherein 2D common images corresponding to the one or more common clusters are encoded only once.
9 . The device of claim 8 , wherein the clustering is further based on semantics associated with points of the 3D scene, on color of the points of the 3D scene, or on motion of the points of the 3D scene.
10 . The device of claim 8 , wherein the processor is further configured for encoding metadata comprising:
a list of viewing boxes defined in the 3D scene and a list of common clusters for the 3D scene; and
for each viewing box, a list of sets of clusters representative of the 3D scene for the respective viewing box and, for each set of clusters associated with the viewing box, identifiers of the common clusters, and the list of clusters other than the common clusters.
11 . A device for decoding a 3D scene comprising a memory associated with a processor configured for:
decoding a first set of first 2D images from a first data stream and a second 2D image from each data stream of a set of distinct data streams, the first 2D images being representative of a projection according to first projection parameters of at least one cluster of points of the 3D scene visible from a first set of points of view encompassed in a first viewing box defined in the 3D scene, the second 2D images being representative of a projection according to second projection parameters of at least one cluster of points of the 3D scene visible from a second set of points of view encompassed in a second viewing box defined in the 3D scene different from the first viewing box, wherein 2D common images corresponding to common clusters to the first and second viewing boxes are decoded only once, and the points of the 3D scene being clustered into a plurality of clusters according to a depth range of the points of the 3D scene, wherein the plurality of clusters comprises at least a background cluster and a foreground cluster; and
un-projecting pixels of the first 2D images according to the first projection parameters and to the first set of points of view and un-projecting pixels of the second 2D images according to the second projection parameters and to the second set of points of view.
12 . The device of claim 11 , further the processor is further configured for obtaining metadata comprising:
a list of viewing boxes defined in the 3D scene and a list of common clusters for the 3D scene; and
for each viewing box, a list of sets of clusters representative of the 3D scene for the respective viewing box and, for each set of clusters associated with the viewing box, identifiers of the common clusters, and the list of clusters other than the common clusters; and
decoding 2D images from data streams comprising clusters of 3D points visible from a current point.