Processing for encoding screen content video using bit allocation
A method for processing a screen content video. The screen content video includes a plurality of frames each including a plurality of coding tree units and a plurality of coding units in each of the coding tree units. The method includes performing a coding-tree-unit-based analysis operation on the screen content video to determine content information associated with the screen content video, and performing a rate control operation on the screen content video based on the determined content information to encoding of the screen content video. The content information includes content complexity information associated with the screen content video and temporal importance information associated with the screen content video.
1 . A method for encoding a screen content video, the screen content video comprising a plurality of frames each including a plurality of coding tree units (CTUs) and a plurality of coding units in each of the coding tree units, the method comprising:
performing a coding-tree-unit-based analysis operation on the screen content video to determine content information associated with the screen content video, the content information including content complexity information and temporal importance information each associated with the screen content video, wherein the content complexity information refers to a measure of spatial characteristics of the screen content in each frame;
modelling a rate-distortion relationship of the screen content video by incorporating the content complexity information into each of a rate model and a distortion model, wherein the modelling is performed at both frame level and CTU level for each frame and each CTU, and wherein the rate model and the distortion model are two separate models each dependent on at least the content complexity and a quantization stepsize;
performing a rate control operation on the screen content video, wherein the rate control operation includes steps of performing bits allocation using a cost function incorporating the temporal importance information to weight distortion predicted by the distortion model, as well as the rate model and the distortion model to estimate rate and distortion for optimization of the bits allocation,
wherein the temporal importance information represents a measure of distortion impact of a coding unit, a coding tree unit, or a frame on a total distortion of a group of pictures, derived recursively through propagation of distortion based on inter-frame prediction and intra block copy prediction; and
and deriving coding parameters according to allocated bits resulting from the bits allocation step.
2 . The method of claim 1 , wherein the content complexity information associated with the screen content video comprises content complexity measures for each of the coding units; and
wherein the temporal importance information comprises temporal importance measures for each of the coding units.
3 . The method of claim 2 , wherein the coding-tree-unit-based analysis operation comprises:
processing the screen content video to perform inter prediction, intra prediction, and intra block copy prediction.
4 . The method of claim 3 , wherein the coding-tree-unit-based analysis operation comprises:
determining the content complexity measures based on Hadamard transform of residuals of the intra prediction, the inter prediction, and/or the intra block copy prediction.
5 . The method of claim 4 , wherein the content complexity measures are based on:
C
=
∑
k
❘
"\[LeftBracketingBar]"
HAD
k
❘
"\[RightBracketingBar]"
W
·
H
Where C denotes a content complexity measure, HAD k denotes a sample of Hadamard-transformed prediction residual at position k within a coding unit, W and H are width and height of a corresponding one of the frame.
6 . The method of claim 4 , wherein the coding-tree-unit-based analysis operation comprises:
determining the temporal importance measures based on the recursive propagation process takes into account the content complexity measures associated with the coding units.
7 . The method of claim 2 , wherein each of the content complexity measures is determined by a sum of absolute transformed difference (SATD).
8 . The method of claim 1 , wherein the rate and distortion model comprises one or more rate models and one or more distortion models.
9 . The method of claim 8 , wherein each of the one or more rate models is modelled based on R=α·C β ·QS γ , where R is rate, C is content complexity measure, QS is quantization stepsize, and α, β, γ are model parameters; and
wherein each of the one or more distortion models is modelled based on D=μ·C η ·QS ϵ , where D is distortion, C is content complexity measure, QS is quantization stepsize, and μ, η, ϵ are model parameters.
10 . The method of claim 9 , wherein the bits allocation step comprises:
group-of-pictures-level bit allocation;
frame-level bit allocation; and
coding-tree-unit-level bit allocation.
11 . The method of claim 10 , wherein the rate control operation further comprises:
determining the coding parameters associated with each of the frames based on the allocated bits obtained in the frame-level bit allocation and the rate and distortion models; and
determining the coding parameters associated with each of the coding tree units based on the allocated bits obtained in the coding-tree-unit-level bit allocation and the rate and distortion models.
12 . The method of claim 11 ,
wherein the coding parameters associated with each of the frames comprise quantization parameters and Lagrangian multipliers λ associated with each of the frames; and
wherein the coding parameters associated with each of the coding tree units comprise quantization parameters and Lagrangian multipliers λ associated with each of the coding tree units.
13 . The method of claim 12 , wherein the Lagrangian multipliers λ associated with each of the frames are determined based on
λ
=
x
·
C
y
QS
z
,
where
x
=
-
μϵ
α
γ
,
y
=
η
-
β
,
z
=
ϵ
-
γ
.
14 . The method of claim 12 , wherein the Lagrangian multipliers λ associated with each of the coding tree units are determined based on
λ
=
x
·
C
y
Q
S
z
,
where
x
=
-
μϵ
α
γ
,
y
=
η
-
β
,
z
=
ϵ
-
γ
.
15 . The method of claim 8 , wherein the one or more rate models comprise a frame-level rate model and a coding-tree-unit-level rate model; and wherein the one or more distortion models comprises a frame-level distortion model, and a coding-tree-unit-level distortion model.
16 . The method of claim 15 ,
wherein the frame-level rate model and the coding-tree-unit-level rate model are each modelled based on R=α·C β ·QS γ , where R is rate, C is content complexity measure, QS is quantization stepsize, and α, β, γ are model parameters; and
wherein the frame-level distortion model and the coding-tree-unit-level distortion model are each modelled based on D=μ·C η ·QS ϵ , where D is distortion, C is content complexity measure, QS is quantization stepsize, and μ, η, ϵ are model parameters.
17 . The method of claim 9 , further comprising: encoding each of the frames and/or each of the coding tree units of the screen content video based on the rate control operation to facilitate generation of a bitstream of the screen content video.
18 . The method of claim 17 , further comprising: updating the model parameters in the rate and distortion models after encoding of each of the frames and/or each of the coding tree units.
19 . The method of claim 1 , comprising a further step of incorporating screen content coding tools in the coding-tree-unit-based analysis operation.
20 . The method of claim 19 , wherein the screen content coding tools are selected from any one of intra block copy (IBC), palette mode, adaptive color Transform (ACT), transform skip with residual coding (TSRC), block-based differential pulse-coded modulation (BDPCM), or a combination thereof.
21 . A system for encoding a screen content video, the screen content video comprising a plurality of frames each including a plurality of coding tree units and a plurality of coding units in each of the coding tree units, the system comprising:
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
performing a coding-tree-unit-based analysis operation on the screen content video to determine content information associated with the screen content video, the content information including content complexity information and temporal importance information each associated with the screen content video, wherein the content complexity information refers to a measure of spatial characteristics of the screen content in each frame;
modelling a rate-distortion relationship of the screen content video by incorporating the content complexity information into each of a rate model and a distortion model, wherein the modelling is performed at both frame level and CTU level for each frame and each CTU, and wherein the rate model and the distortion model are two separate models each dependent on at least the content complexity and a quantization stepsize;
performing a rate control operation on the screen content video, wherein the rate control operation includes steps of performing bits allocation using a cost function incorporating the temporal importance information to weight distortion predicted by the distortion model, as well as the rate model and the distortion model to estimate rate and distortion for optimization,
wherein the temporal importance information represents a measure of distortion impact of a coding unit, a coding tree unit, or a frame on a total distortion of a group of pictures, derived recursively through propagation of distortion based on inter-frame prediction and intra block copy prediction; and
deriving coding parameters according to allocated bits resulted from the bits allocation step.
22 . A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for encoding a screen content video,
wherein the screen content video comprises a plurality of frames each including a plurality of coding tree units and a plurality of coding units in each of the coding tree units, and
wherein the instructions for processing a screen content video comprise instructions for:
performing a coding-tree-unit-based analysis operation on the screen content video to determine content information associated with the screen content video, the content information including content complexity information and temporal importance information each associated with the screen content video, wherein the content complexity information refers to a measure of spatial characteristics of the screen content in each frame;
modelling a rate-distortion relationship of the screen content video by incorporating the content complexity information into each of a rate model and a distortion model, wherein the modelling is performed at both frame level and CTU level for each frame and each CTU, and wherein the rate model and the distortion model are two separate models each dependent on at least the content complexity and a quantization stepsize;
performing a rate control operation on the screen content video, wherein the rate control operation includes steps of performing bits allocation using a cost function incorporating the temporal importance information to weight distortion predicted by the distortion model, as well as the rate model and the distortion model to estimate rate and distortion for optimization of the bits allocation,
wherein the temporal importance information represents a measure of distortion impact of a coding unit, a coding tree unit, or a frame on a total distortion of a group of pictures, derived recursively through propagation of distortion based on inter-frame prediction and intra block copy prediction; and
deriving coding parameters according to allocated bits resulted from the bits allocation step.