Bit stream structure for compressed point cloud data
A system comprises an encoder configured to compress attribute information and/or spatial information for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud. In some embodiments, a bit stream structure may be used to communicate compressed point cloud data. The bit stream structure may include point cloud compression network abstraction layer (PCCNAL) units that enable use of groups of frames (GOFs), frame, and sub-frame signaling of patch information. Such a bit stream structure may permit low delay streaming and random access reconstruction of point clouds amongst other applications.
1. A non-transitory, computer-readable, storage medium storing program instructions, that when executed using one or more processors, cause the one or more processors to:
receive three-dimensional (3D) content to be compressed;
generate, for the three-dimensional content:
attribute image frames comprising attribute patches for the 3D content packed into respective two-dimensional (2D) frames;
geometry image frames comprising geometry patches for the 3D content packed into respective 2D frames; and
encode the generated attribute image frames and the generated geometry image frames,
wherein:
at least some coding parameters used to encode the attribute image frames are different than at least some coding parameter used to encode the geometry image frames, wherein the differing coding parameters comprise one or more of:
differing bit-depths used to encode the attribute image frames and the geometry image frames; or
differing sizing parameters used to encode the attribute image frames and the geometry image frames; and
an encoded bitstream, resulting from the encoding, comprises one or more network abstraction layer (NAL) units that indicate a first one of the differing coding parameters to be used to decode the attribute image frames and a second one of the differing coding parameters to be used to decode the geometry image frames.
2. The non-transitory, computer-readable, storage medium of claim 1 , wherein the one or more NAL units indicate:
a first frame height or frame width for a portion of a given one of the 2D frames that corresponds to one or more of the attribute image frames; and
a second frame height or frame width for a portion of a given one of the 2D frames that corresponds to one or more of the geometry image frames.
3. The non-transitory, computer-readable, storage medium of claim 1 , wherein the one or more NAL units indicate:
a resizing parameter that indicates a re-sizing between a video frame size and an image frame size of one or more of the attribute image frames or one or more of the geometry image frames.
4. The non-transitory, computer-readable, storage medium of claim 1 , wherein the one or more NAL units indicate:
a color format to be used for decoding a respective set of image frames corresponding to one or more of the attribute image frames or one or more of the geometry image frames.
5. The non-transitory, computer-readable, storage medium of claim 1 , wherein the one or more NAL units indicate:
a first video bit-depth of geometry values encoded for patch images of one or more of the geometry image frames;
a first corresponding reconstructed bit-depth for the geometry values to be used to reconstruct the 3D content;
a second video bit-depth of attribute values encoded for patch images of one or more of the attribute image frames; and
a second corresponding reconstructed bit-depth for the attribute values to be used to reconstruct the 3D content.
6. The non-transitory, computer-readable, storage medium of claim 1 , wherein the one or more NAL units indicate:
an internal bit-depth used to determine reconstructed attribute values or reconstructed geometry values based on attribute values or geometry values included in patch images of one or more of the attribute image frames or one or more of the geometry image frames.
7. The non-transitory, computer-readable, storage medium of claim 1 , wherein the one or more NAL units indicate:
sequence parameter set (SPS) information for a sequence of the video encoded attribute and/or geometry image frames; or
frame parameter set (FPS) information for individual ones of the video encoded attribute or geometry image frames.
8. The non-transitory, computer-readable, storage medium of claim 7 , wherein:
the encoded bitstream comprises one or more NAL units indicating sequence parameter set (SPS) information and one or more NAL units indicating frame parameter set (FPS) information; and
the FPS information indicates one or more parameters, to be applied for a given attribute or geometry image frame, that differ from one or more parameters specified in the sequence parameter set (SPS) information.
9. The non-transitory, computer-readable, storage medium of claim 7 , wherein the encoded bitstream comprises:
a first NAL unit indicating a sequence parameter set to be applied to a first set of image frames;
a set of the geometry image frames, included in the first set, corresponding to the first NAL unit indicating the sequence parameter set;
a set of the attribute image frames, included in the first set, corresponding to the first NAL unit indicating the sequence parameter set; and
a set of occupancy image frames, included in the first set, corresponding to the first NAL unit indicating the sequence parameter set.
10. The non-transitory, computer-readable, storage medium of claim 9 , wherein, the video image frames of the respective sets of the first set (geometry, attribute, and occupancy) are signaled together.
11. The non-transitory, computer-readable, storage medium of claim 9 , wherein, the video image frames of the respective sets of the first set (geometry, attribute, and occupancy) are interleaved.
12. A non-transitory, computer-readable, storage medium storing program instructions, that when executed using one or more processors, cause the one or more processors to:
receive a bit stream comprising:
attribute image frames comprising attribute patches for three-dimensional (3D) content packed into respective two-dimensional (2D) frames;
geometry image frames comprising geometry patches for the 3D content packed into respective 2D frames; and
one or more network abstraction layer (NAL) units that indicate differing coding parameters to be used to decode the attribute image frames and the geometry image frames, wherein the differing coding parameters comprise a first set of coding parameter to be used to decode the attribute image frames and a second set of coding parameters to be used to decode the geometry image frames, wherein the differing coding parameters comprise one or more of:
differing bit-depths used to encode the attribute image frames and the geometry image frames; or
differing sizing parameters used to encode the attribute image frames and the geometry image frames;
decode the attribute image frames using the first set of coding parameters;
decode the geometry image frames using the second set of coding parameters, wherein the first set and the second set of coding parameters are determined based on the one or more NAL units, and wherein at least one coding parameter differs between the first set and the second set of coding parameters; and
reconstruct three-dimensional content using attribute information and geometry information from the decoded attribute image frame and the decoded geometry image frames.
13. The non-transitory, computer-readable, storage medium of claim 12 , wherein the one or more NAL units indicate:
a frame height or frame width for a portion of a given one of the 2D frames that corresponds to one or more of the attribute image frames or one or more of the geometry image frames.
14. The non-transitory, computer-readable, storage medium of claim 12 , wherein the one or more NAL units indicate:
a resizing parameter that indicates a re-sizing between a video frame size and an image frame size of one or more of the attribute image frames or one or more of the geometry image frames.
15. The non-transitory, computer-readable, storage medium of claim 12 , wherein the one or more NAL units indicate:
a video bit-depth of geometry values or attribute values encoded for patch images of one or more of the attribute image frames or one or more of the geometry image frames; and
a corresponding reconstructed bit-depth for the geometry values or attribute values to be used to reconstruct the 3D content.
16. The non-transitory, computer-readable, storage medium of claim 12 , wherein the received bitstream comprises:
a first NAL unit indicating a sequence parameter set to be applied to a first set of image frames;
a set of the geometry image frames, included in the first set, corresponding to the first NAL unit indicating the sequence parameter set;
a set of the attribute image frames, included in the first set, corresponding to the first NAL unit indicating the sequence parameter set; and
a set of occupancy image frames, included in the first set, corresponding to the first NAL unit indicating the sequence parameter set.
17. The non-transitory, computer-readable, storage medium of claim 16 , wherein, the video image frames of the respective sets of the first set (geometry, attribute, and occupancy) are interleaved.
18. A device, comprising:
a memory storing program instructions; and
one or more processors, wherein the program instructions, when executed using the one or more processors, cause
the one or more processors to:
receive a bit stream comprising:
attribute image frames comprising attribute patches for three-dimensional (3D) content packed into respective two-dimensional (2D) frames;
geometry image frames comprising geometry patches for the 3D content packed into respective 2D frames; and
one or more network abstraction layer (NAL) units that indicate differing coding parameters to be used to decode the attribute image frames and the geometry image frames, wherein the differing coding parameters comprise a first set of coding parameter to be used to decode the attribute image frames and a second set of coding parameters to be used to decode the geometry image frames, wherein the differing coding parameters comprise one or more of:
differing bit-depths used to encode the attribute image frames and the geometry image frames; or
differing sizing parameters used to encode the attribute image frames and the geometry image frames;
decode the attribute image frames using the first set of coding parameters;
decode the geometry image frames using the second set of coding parameters, wherein the first set and the second set of coding parameters are determined based on the one or more NAL units, and wherein at least one coding parameter differs between the first set and the second set of coding parameters; and
reconstruct three-dimensional content using attribute information and geometry information from the decoded attribute image frame and the decoded geometry image frames.
19. The device of claim 18 , wherein the one or more NAL units indicate one or more of:
a frame height or frame width for a portion of a given one of the 2D frames that corresponds to one or more of the attribute image frames or one or more of the geometry image frames;
a resizing parameter that indicates a re-sizing between a video frame size and an image frame size of one or more of the attribute image frames or one or more of the geometry image frames;
a color format to be used for decoding a respective set of image frames corresponding to one or more of the attribute image frames or one or more of the geometry image frames;
a video bit-depth of geometry values or attribute values encoded for patch images of one or more of the attribute image frames or one or more of the geometry image frames and a corresponding reconstructed bit-depth for the geometry values or attribute values to be used to reconstruct the 3D content; or
an internal bit-depth used to determine reconstructed attribute values or reconstructed geometry values based on attribute values or geometry values included in patch images of one or more of the attribute image frames or one or more of the geometry image frames.
20. The device of claim 18 , further comprising:
a display,
wherein the program instructions, when executed using the one or more processors, further cause the one or more processors to:
render the reconstructed 3D content for viewing on the display.