Processing of filler data units in video streams

1. A method of video processing, comprising:

performing a conversion between a video and a bitstream of the video according to a rule,

wherein the rule specifies that a sub-bitstream extraction process is implemented to generate a sub-bitstream for decoding,

wherein the sub-bitstream extraction process is configured to extract, from the bitstream, the sub-bitstream with a target highest temporal identifier,

wherein the rule specifies that, during the sub-bitstream extraction process, upon removing a video coding layer (VCL) network abstraction layer (NAL) unit, filler data units and filler supplemental enhancement information (SEI) messages in SEI NAL units that are associated with the VCL NAL unit are also removed, and

wherein the bitstream conforms to a format rule that specifies that a SEI NAL unit that comprises a SEI message with a filler payload does not include SEI messages with a payload different from the filler payload.

2. The method of claim 1 , wherein the VCL NAL unit is removed based on an identifier of a layer that the VCL NAL unit belongs to.

3. The method of claim 2 , wherein the identifier is nuh_layer_id.

4. The method of claim 1 , wherein the bitstream conforms to a format rule that specifies that the SEI NAL unit that comprises the SEI message with the filler payload is configured to exclude another SEI message with any other payload.

5. The method of claim 1 , wherein the conversion comprises decoding the video from the bitstream.

6. The method of claim 1 , wherein the conversion comprises encoding the video into the bitstream.

7. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

perform a conversion between a video and a bitstream of the video according to a rule,

wherein the rule specifies that a sub-bitstream extraction process is implemented to generate a sub-bitstream for decoding,

wherein the sub-bitstream extraction process is configured to extract, from the bitstream, the sub-bitstream with a target highest temporal identifier,

wherein the rule specifies that, during the sub-bitstream extraction process, upon removing a video coding layer (VCL) network abstraction layer (NAL) unit, filler data units and filler supplemental enhancement information (SEI) messages in SEI NAL units that are associated with the VCL NAL unit are also removed, and

wherein the bitstream conforms to a format rule that specifies that a SEI NAL unit that comprises a SEI message with a filler payload does not include SEI messages with a payload different from the filler payload.

8. The apparatus of claim 7 , wherein the VCL NAL unit is removed based on an identifier of a layer that the VCL NAL unit belongs to.

9. The apparatus of claim 8 , wherein the identifier is nuh_layer_id.

10. The apparatus of claim 7 , wherein the bitstream conforms to a format rule that specifies that the SEI NAL unit that comprises the SEI message with the filler payload is configured to exclude another SEI message with any other payload.

11. A non-transitory computer-readable storage medium storing instructions that cause a processor to:

perform a conversion between a video and a bitstream of the video according to a rule,

wherein the rule specifies that a sub-bitstream extraction process is implemented to generate a sub-bitstream for decoding,

wherein the sub-bitstream extraction process is configured to extract, from the bitstream, the sub-bitstream with a target highest temporal identifier,

wherein the rule specifies that, during the sub-bitstream extraction process, upon removing a video coding layer (VCL) network abstraction layer (NAL) unit, filler data units and filler supplemental enhancement information (SEI) messages in SEI NAL units that are associated with the VCL NAL unit are also removed, and

wherein the bitstream conforms to a format rule that specifies that a SEI NAL unit that comprises a SEI message with a filler payload does not include SEI messages with a payload different from the filler payload.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the VCL NAL unit is removed based on an identifier of a layer that the VCL NAL unit belongs to.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the identifier is nuh_layer_id.

14. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

generating the bitstream of the video according to a rule,

wherein the rule specifies that a sub-bitstream extraction process is implemented to generate a sub-bitstream for decoding, wherein the sub-bitstream extraction process is configured to extract, from the bitstream, the sub-bitstream with a target highest temporal identifier,

wherein the rule specifies that, during the sub-bitstream extraction process, upon removing a video coding layer (VCL) network abstraction layer (NAL) unit, filler data units and filler supplemental enhancement information (SEI) messages in SEI NAL units that are associated with the VCL NAL unit are also removed, and

wherein the bitstream conforms to a format rule that specifies that a SEI NAL unit that comprises a SEI message with a filler payload does not include SEI messages with a payload different from the filler payload.

15. The non-transitory computer-readable recording medium of claim 14 , wherein the VCL NAL unit is removed based on an identifier of a layer that the VCL NAL unit belongs to.

16. The non-transitory computer-readable recording medium of claim 15 , wherein the identifier is nuh_layer_id.

17. The apparatus of claim 7 , wherein the conversion comprises decoding the video from the bitstream.

18. The apparatus of claim 7 , wherein the conversion comprises encoding the video into the bitstream.

19. The non-transitory computer-readable storage medium of claim 11 , wherein the bitstream conforms to a format rule that specifies that the SEI NAL unit that comprises the SEI message with the filler payload is configured to exclude another SEI message with any other payload.

20. The non-transitory computer-readable recording medium of claim 14 , wherein the bitstream conforms to a format rule that specifies that the SEI NAL unit that comprises the SEI message with the filler payload is configured to exclude another SEI message with any other payload.