Signaling displacement data for video-based mesh coding
A system comprises an encoder configured to compress and encode data for a three-dimensional mesh. To compress the three-dimensional mesh, the encoder determines displacements to be applied to sub-division locations of a base mesh. The displacement values may be signaled in their own sub-bitstream, e.g., a dedicated displacement data sub-bitstream, or may be signaled, at least in part, in an atlas data sub-bitstream that includes patch data. In some embodiments, the displacements may also be signaled, at least in part, in a video sub-bitstream or in a base-mesh sub-bitstream.
1 . A non-transitory, computer-readable, storage medium storing program instructions, that when executed using one or more computing devices, cause the one or more computing devices to:
compress visual volumetric content using a dynamic mesh compression algorithm, wherein compressing the visual volumetric comprises:
compressing a base mesh to generate a base mesh sub-bitstream to be included in a bitstream for the compressed visual volumetric content;
determining displacement information for displacements that are to be applied to sub-division locations of the base mesh; and
compressing attribute information, wherein the compressed attribute information is to be included in a video sub-bitstream of the bitstream for the compressed visual volumetric content; and
provide the bitstream for the compressed visual volumetric content, wherein the displacement information is signaled, at least in part, in the bitstream in a sub-bitstream other than the video sub-bitstream, and wherein the sub-bitstream used to signal the displacement information uses a syntax that is independent of other syntaxes used for other ones of the sub-bitstreams of the bitstream.
2 . The non-transitory, computer-readable, storage medium of claim 1 , wherein the displacement information is signaled using a displacement data unit syntax.
3 . The non-transitory, computer-readable, storage medium of claim 1 , wherein patch data units for the displacement information are signaled in an atlas data sub-bitstream of the bitstream for the compressed visual volumetric content.
4 . The non-transitory, computer-readable, storage medium of claim 3 , wherein the displacement information is signaled, at least in part, in the sub-bitstream used to signal the displacement information using patch data units that correspond to the patch data units used in the atlas data sub-bitstream.
5 . The non-transitory, computer-readable, storage medium of claim 2 , wherein the displacement information is signaled, at least in part, using a sequence parameter set header of the displacement data unit syntax.
6 . The non-transitory, computer-readable, storage medium of claim 5 , wherein the displacement information is signaled, at least in part, using a frame parameter set header of the displacement data unit syntax.
7 . The non-transitory, computer-readable, storage medium of claim 1 , wherein the syntax used to signal displacement information is a displacement data unit syntax, and wherein the displacement data unit syntax is configured to enable partial decoding or random decoding of the displacement information.
8 . The non-transitory, computer-readable, storage medium of claim 1 , wherein the displacement information is further signaled, at least in part in the video sub-bitstream,
wherein a flag in a portion of the displacement information signaled in the sub-bitstream other than the video sub-bitstream is used to signal portions of the displacement information signaled in the video sub-bitstream.
9 . The non-transitory, computer-readable storage medium of claim 8 , wherein the portion of the displacement information signaled in the video sub-bitstream is signaled such that:
a first displacement component is signaled in a first color plane of the video sub-bitstream;
a second displacement component is signaled in a second color plane of the video sub-bitstream; and
a third displacement component is signaled in a third color plane of the video sub-bitstream,
wherein one or more resolutions, at which the first, the second, and the third components of the displacement information are signaled, are adjusted to account for sub-sampling between the first, the second, and the third color planes of the video sub-bitstream.
10 . The non-transitory, computer-readable, storage medium of claim 8 , wherein the portion of the displacement information signaled in the video sub-bitstream is signaled using a single color plane of the video sub-bitstream.
11 . The non-transitory, computer-readable storage medium of claim 8 , wherein the portion of the displacement information signaled in the video sub-bitstream is signaled such that:
displacement information for different sub-meshes of the reconstructed mesh are signaled using starting points in different tiles of image frames of the video sub-bitstream.
12 . A non-transitory, computer-readable, storage medium storing program instructions, that when executed using one or more computing devices, cause the one or more computing devices to:
receive a bitstream representing a compressed version of visual volumetric content, the bitstream comprising:
a base mesh sub-bitstream;
a video sub-bitstream; and
displacement information for displacements that are to be applied to sub-division locations of a base mesh signaled in the base mesh sub-bitstream,
wherein the displacement information is signaled, at least in part, in the bitstream in a sub-bitstream other than the video sub-bitstream, and wherein the sub-bitstream used to signal the displacement information uses a syntax that is independent of other syntaxes used for other ones of the sub-bitstreams of the bitstream; and
reconstruct a mesh of the visual volumetric content, wherein to reconstruct the mesh the program instructions cause the one or more computing devices to:
sub-divide edges of the base mesh to generate the sub-division locations;
parse the bitstream to identify the displacement information; and
apply the displacements indicated in the displacement information to the sub-division locations of the base mesh.
13 . The non-transitory, computer-readable, storage medium of claim 12 , wherein the displacement information is signaled using a displacement data unit syntax in a displacement sub-bitstream, which is a separate sub-bitstream from the video sub-bitstream, the base mesh sub-bitstream, and an atlas data sub-bitstream.
14 . The non-transitory, computer-readable, storage medium of claim 12 , wherein patch data units for the displacement information are signaled in an atlas data sub-bitstream of the bitstream.
15 . The non-transitory, computer-readable, storage medium of claim 12 , wherein the syntax used to signal the displacement information is a displacement data unit syntax, and wherein the displacement data unit syntax is configured to enable partial decoding or random decoding of the displacement information.
16 . The non-transitory, computer-readable, storage medium of claim 12 , wherein the displacement information is further signaled, at least in in part, in the video sub-bitstream.
17 . The non-transitory, computer-readable, storage medium of claim 16 , wherein a flag in a portion of the displacement information signaled in the sub-bitstream other than the video sub-bitstream is used to signal portions of the displacement information signaled in the video sub-bitstream.
18 . A device, comprising:
a memory storing program instructions; and
one or more processors, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
receive a bitstream representing a compressed version of visual volumetric content, the bitstream comprising:
a base mesh sub-bitstream;
a video sub-bitstream; and
displacement information for displacements that are to be applied to sub-division locations of a base mesh signaled in the base mesh sub-bitstream,
wherein the displacement information is signaled, at least in part, in the bitstream in a sub-bitstream other than the video sub-bitstream, and wherein the sub-bitstream used to signal the displacement information uses a syntax that is independent of other syntaxes used for other ones of the sub-bitstreams of the bitstream;
reconstruct a mesh of the visual volumetric content, wherein reconstructing the mesh comprises:
sub-dividing edges of the base mesh to generate the sub-division locations;
parsing the bitstream to identify the displacement information; and
applying the displace indicated in the displacement information to the sub-division locations of the base mesh.
19 . The device of claim 18 , wherein patch data units for the displacement information are signaled, at least in part, in an atlas data sub-bitstream of the bitstream.
20 . The device of claim 18 , wherein the displacement information is signaled using a displacement data unit syntax.