Neural network-based video compression with bit allocation

1. A method of video processing, comprising:

determining, for a conversion between a current video unit of a current video frame of a video and a bitstream of the video, a target bitrate for the current video unit that is based on a rate distortion function in which a rate portion is weighted using lambda, wherein the lambda is a rational number, wherein the lambda is adaptively adjusted for each video unit of the video, and wherein the lambda is adjusted using multiple neural networks, and different neural networks in the multiple neural networks are trained for determining values of lambda for different video units; and

performing the conversion based on the determining.

2. The method of claim 1 , wherein lambda for the current video unit is scaled based on at least one scaling factor, and the at least one scaling factor depends on at least one of a scaling factor set, the target bitrate, a coding order, a temporal layer identifier for the current video unit, a location of the current video unit, and color component.

3. The method of claim 1 , wherein lambda for the current video unit is based on the target bitrate for the current video unit.

4. The method of claim 1 , wherein the current video unit is a coding tree unit row or a coding tree unit or a coding tree block or a coding unit or a prediction unit or a transform unit or a coding block or a transform block or a virtual pipeline data unit or a picture or a slice or a tile or a subpicture or a subset of pixels within a coding tree unit.

5. The method of claim 1 , wherein lambda for the current video unit is based on a temporal layer identifier for the current video frame.

6. The method of claim 5 , wherein the lambda for the current video unit is based on a structure of a group of pictures (GOP) of the video, and the structure of the GOP includes a size of the GOP or a reference picture setting for the GOP.

7. The method of claim 1 , wherein lambda for a current region within the current video unit is obtained by applying a scaling factor to a base lambda value, or adjusting a quantization step based on related information, and the lambda is adaptively adjusted for each video region of the current video unit of the video.

8. The method of claim 7 , wherein the scaling factor is determined on the fly or selected from a candidate set.

9. The method of claim 7 , wherein the related information is signaled for one or multiple or all spatial regions for a picture of the video, the related information comprises characteristics of a spatial location for the current region, and the characteristics are measured in a sample space or perceptual space.

10. The method of claim 1 , wherein the current video frame is predictively coded using N reference frames, wherein N is an integer greater than 2, and N depends on a temporal layer identifier for the current video frame or a coding order of the current video frame.

11. The method of claim 10 , wherein, in case that a number of available reference frames is greater than C, the N reference frames correspond to C temporally closest reference frames to the current video frame, with C being a maximum number of reference frames for the video.

12. The method of claim 10 , wherein a neural network determines reference frames used for different video regions of the current video unit.

13. The method of claim 1 , wherein the current video frame is coded in the bitstream using a residual frame that is coded using a residual that is based on an interpolated frame determined from multiple video frames of the video.

14. The method of claim 13 , wherein the interpolated frame is concatenated with the residual frame for the conversion.

15. The method of claim 13 , wherein individual features are extracted from the interpolated frame and the residual frame and fused together during the conversion.

16. The method of claim 1 , wherein the conversion includes encoding the current video unit into the bitstream.

17. The method of claim 1 , wherein the conversion includes decoding the current video unit from the bitstream.

18. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

determine, for a conversion between a current video unit of a current video frame of a video and a bitstream of the video, a target bitrate for the current video unit that is based on a rate distortion function in which a rate portion is weighted using lambda, wherein the lambda is a rational number, wherein the lambda is adaptively adjusted for each video unit of the video, and wherein the lambda is adjusted using multiple neural networks, and different neural networks in the multiple neural networks are trained for determining values of lambda for different video units; and

perform the conversion based on the determining.

19. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

determining, for a current video unit of a current video frame of a video, a target bitrate for the current video unit that is based on a rate distortion function in which a rate portion is weighted using lambda, wherein the lambda is a rational number, wherein the lambda is adaptively adjusted for each video unit of the video, and wherein the lambda is adjusted using multiple neural networks, and different neural networks in the multiple neural networks are trained for determining values of lambda for different video units; and

generating the bitstream based on the determining.