Recursive segment to scene segmentation for cloud-based coding of HDR video
In a cloud-based system for encoding high dynamic range (HDR) video, each node receives a video segment and bumper frames. Each segment is subdivided into primary scenes and secondary scenes to derive scene-based forward reshaping functions that minimize the amount of reshaping-related metadata when coding the video segment, while maintaining temporal continuity among scenes processed by multiple nodes. Methods to generate scene-based forward and backward reshaping functions to optimize video coding and improve the coding efficiency of reshaping-related metadata are also examined.
1. A method for segmenting a video segment into scenes using a cloud-based system for encoding high dynamic range video, the method comprising:
receiving in a current computing node of the cloud-based system a first video sequence comprising video frames in a high dynamic range;
generating for each video frame in the first video sequence a frame-based forward reshaping function mapping the video frame from the high dynamic range to a second dynamic range lower than the high dynamic range;
generating, using a set of scene cuts for the first video sequence, a set of primary scenes for the first video sequence;
generating a second set of scenes for the first video sequence based on the set of primary scenes, wherein a primary scene belonging to a parent scene with video frames to be coded across the current computing node and a neighbor computing node of the cloud-based system is divided into secondary scenes, wherein, given a primary scene, generating a list of secondary scenes for the primary scene comprises:
initializing a set of secondary scenes and a set of violation scenes based on the set of primary scenes; and
generating one or more sets of smoothness thresholds based on the frame-based forward reshaping functions, wherein generating the one or more smoothness thresholds comprises computing a first set of smoothness thresholds ℑ j DC for each frame i in the first video sequence,
ℑ j DC =χ j −χ j−1 ,
wherein
χ j = 1 / H×WΣ b {tilde over (T)} P F ( b )× h j v ( b ),
wherein T{tilde over ( )}_j{circumflex over ( )}F (b) denotes a frame-based forward reshaping function for frame-j in the first video sequence as a function of input codewords b, h j v (b) denotes a histogram of the j-th frame in the first video sequence, and H and W denote width and height values for the frames in the first video sequence;
generating for each scene in the second set of scenes a scene-based forward reshaping function mapping the video frames in the scene from the high dynamic range to the second dynamic range;
applying the scene-based forward reshaping functions to the video frames in the first video sequence to generate an output video sequence comprising video frames in the second dynamic range; and
compressing the output video sequence to generate a coded bitstream.
2. The method of claim 1 , wherein, given a primary scene, generating a list of secondary scenes for the primary scene further comprises:
repeating until there are no boundary violations:
dividing each scene in the set of violation scenes into two new subscenes;
generating an updated set of violation scenes with an empty set;
generating an updated set of secondary scenes by adding the two new subscenes to the set of secondary scenes;
performing one or more boundary violation checks in the set of secondary scenes using the one or more sets of smoothness thresholds; and
if there is at least one boundary violation between two subscenes in the set of secondary scenes, then adding the two subscenes to the set of violation scenes and continuing subdividing the primary scene with the updated set of violation scenes and the updated set of secondary scenes;
else, signaling there are no boundary violations and outputting the list of secondary scenes.
3. The method of claim 1 , wherein generating, using a set of scene cuts for the first video sequence, a set of primary scenes comprises:
accessing a first set of scene cuts for the first video sequence from a file;
generating a second set of scene cuts for the first video sequence using an automatic scene-change detector;
generating a final set of scene cuts based on an intersection of the first set of scene cuts and the second set of scene cuts; and
generating the set of primary scenes using the final set of scene cuts.
4. The method of claim 1 , wherein given a scene P g in the set of violation scenes, the scene is split in frame location C S , wherein:
if the scene P g comprises a primary scene which is part of a parent scene with frames to be processed in a computing node of the cloud-based system prior to the current computing node, then
C S =C 0 +B,
wherein, C 0 denotes the first frame in the first video sequence and B denotes a number of bumper frames shared by two neighboring computing nodes of the cloud-based system;
else, if the scene P g comprises a primary scene which is part of a parent scene with frames to be processed in a computing node of the cloud-based system subsequent to the current computing node, then
C S =C L−1 −B,
wherein C L−1 denotes the last frame in the first video sequence;
else, if the scene P g comprises a secondary scene
C
s
=
min
j
∑
b
❘
"\[LeftBracketingBar]"
T
˜
P
g
F
(
b
)
-
T
˜
j
F
(
b
)
❘
"\[RightBracketingBar]"
,
wherein T{tilde over ( )}_j{circumflex over ( )}F (b) denotes a frame-based forward reshaping function for frame-j in the first video sequence as a function of input codewords b, and {tilde over (T)} P g F (b) denotes an average of frame-based forward reshaping functions for frames related to the scene P g .
5. The method of claim 1 , further comprising computing a second set of smoothness thresholds ℑ j FLUT ,
ℑ j FLUT =max(α×max({| {tilde over (T)} j F ( b )− {tilde over (T)} j−1 F ( b )∥∀ b }),β),
wherein α and β are constants.
6. The method of claim 5 , wherein for a secondary scene P g−1 starting at frame C g−1 and a secondary scene P g starting at frame C g , performing the one or more boundary violation checks between the two scenes comprises:
testing if
|Δ C g |>|ℑ C g DC |+ω
is true, and if true, declaring a boundary violation, wherein w is a constant, and
Δ C g =λ C g −λ C g −1 ,
wherein, for frame j within the scene P g
λ
j
=
1
H
×
W
∑
b
T
˜
P
g
F
(
b
)
×
h
j
v
(
b
)
,
∀
j
∈
[
C
g
,
C
g
+
1
-
1
]
,
and T{tilde over ( )}_(P_g){circumflex over ( )}F (b) denotes an average of frame-based forward reshaping functions in the secondary scene P g and neighbor secondary scenes.
7. The method of claim 6 , wherein
T
˜
P
g
F
(
b
)
=
1
Θ
″
-
Θ
′
+
1
∑
j
=
Θ
′
Θ
″
T
˜
j
F
(
b
)
,
Θ
′
=
max
(
C
0
,
C
g
-
θ
)
,
Θ
″
=
min
(
C
L
-
1
,
C
g
+
1
-
1
+
θ
)
,
θ is an integer constant denoting a frame overlap between two subscenes, and C 0 and C L−1 denote the first and last frame in the first video sequence.
8. The method of claim 6 , further comprising testing if
sign(Δ C g )≠sign(ℑ C g DC )
is true, and if true declaring a boundary violation,
wherein for a real number x, sign (x) returns 0 of x=0, 1, if x>0, and −1 if x<0.
9. The method of claim 6 , further comprising testing if
max({\ T {tilde over ( )}_( P _ g ){circumflex over ( )} F ( b )− T {tilde over ( )}_( P _( g− 1)){circumflex over ( )} F ( b )∥∀ b })>\_( C _ g ){circumflex over ( )}FLUT
is true, and if true, declaring a boundary violation.
10. The method of claim 1 , wherein generating for each scene in the second set of scenes a scene-based forward reshaping function comprises:
if a scene in the second set of scenes is a primary scene, then generating the scene-based forward reshaping function for the scene based only on statistical data generated from frames within the scene; else
if a scene in the second set of scenes is a secondary scene, then generating the scene-based forward reshaping function for the scene based on statistics from frames in the scene and frames from neighboring secondary scenes.
11. The method of claim 10 , further comprising:
generating scene-based backward reshaping functions based on the output video sequence, the first video sequence, and the scene-based forward reshaping functions;
generating metadata based on parameters of the scene-based backward reshaping functions; and
outputting an output bitstream comprising the coded bitstream and the metadata.
12. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with claim 1 .
13. An apparatus comprising a processor and configured to perform the method recited in claim 1 .