Tensor-product B-spline prediction for HDR video in mobile applications
Tensor-Product B-splines (TPB) have been shown to improve video quality when used to represent reshaping functions to map reshaped standard dynamic range content into high dynamic range (HDR) content; however, TPB prediction is computationally intensive and may not be supported by legacy devices. Methods and systems for backwards-compatible signaling of TPB-related metadata and a fast TPB prediction method are presented to overcome both of these limitations. Computation overhead for a TPB-based 3D look-up table is reduced by using temporary two-dimensional arrays. A remapping of the most significant bits of a legacy bit-depth parameter allows for backwards compatibility.
1 . A method for decoding high-dynamic range images and video, the method comprising:
receiving an input image represented in an input dynamic range;
receiving metadata for the input image to generate a mapping function to map the input image from the input dynamic range to an output image in an output dynamic range;
parsing a first metadata parameter to identify whether a second metadata parameter is present;
if the second metadata parameter is present, then
generating the mapping function using a non-backward compatible predictor, wherein the non-backward compatible predictor comprises a tensor-product B-spline (TPB) predictor, wherein generating the mapping function using a tensor-product B-spline (TPB) predictor comprises:
accessing TPB metadata within the metadata of the input image, wherein the TPB metadata comprise TPB-related prediction parameters, prediction coefficients, and auxiliary data; and
generating based on the TPB metadata a three dimensional look up table (3D-LUT) to map selected triplets of pixel values in the input dynamic range to corresponding output triplets of pixel values in the output dynamic range,
wherein generating the 3D-LUT involves, for each 3D-LUT index, generating respective basis functions by using the prediction parameters and computing a respective output pixel value by using the prediction coefficients based on the basis functions;
wherein generating the 3D-LUT involves first storing frequently used values of the basis functions in one or more 2D arrays for later computing output pixel values by using the prediction coefficients;
wherein the prediction parameters comprise:
a number of knot points (T) in each of three dimensions of a luma channel and two chroma channels; and
polynomial order (l) values in each of the three dimensions of the luma channel and the two chroma channels;
wherein the prediction coefficients comprise:
prediction coefficients for the luma channel and the two chroma channels; and
wherein the auxiliary data comprise:
a luma-channel basis flag and two chroma-channel basis flags for determining correlation among the prediction coefficients for the luma channel and the two chroma channels;
else generating the mapping function using a backward-compatible predictor supported by legacy devices; and
applying the mapping function to the input image to generate the output image,
wherein the input dynamic range comprises a standard dynamic range and the output dynamic range comprises a high dynamic range.
2 . The method of claim 1 , wherein the first metadata parameter comprises a parameter for determining bit depth information in an enhancement layer of the input image.
3 . The method of claim 1 , wherein identifying whether the second metadata parameter is present comprises:
masking the first metadata parameter to read two separate values, a first value based on bits 0 - 7 of the first metadata parameter and a second value based on bits 8 - 15 of the first metadata parameter; and
identifying whether the second metadata parameter is present based on the second value.
4 . The method of claim 1 , wherein generating the 3D-LUT further comprises:
if all basis flags are 0, then:
generating nine temporary arrays, three temporary arrays for each color channel;
else if the luma-channel basis flag is 0 and one of the two chroma-channel basis flags is 1, then:
generating three temporary arrays for the luma channel and three temporary arrays for the two chroma channels;
else if the luma-channel basis flag is 0 and both of the two chroma-channel basis flags are 1, then:
generating three temporary arrays to be used for all color channels; and
generating the 3D-LUT for each color channel based on a product of the three temporary arrays for that color channel.
5 . The method of claim 4 , wherein generating three temporary arrays for the luma color channel comprises computing arrays A y,y (s ji y , k p y,y ), A y,c0 (s ji c0 , k q y,c0 ) and A y,c1 (s ji c1 , k q y,c1 ), wherein for an L 0 ×L 1 ×L 2 3D-LUT:
A y,y (s ji y , k p y,y ) denotes (T y,y +l y,y −1)×L 0 values of a B j,q y,0 (s ji y , k p y,y , h y,y , l y,y ) B-spline function for input luma pixel values s ji y , for i=0, 1, . . . , L 0 , at uniformly distributed knot points k p y,y at distance h y,y , order l y,y , and a total of T y,y k p y,y knot points;
A y,c0 (s ji c0 , k q y,c0 ) denotes (T y,c0 +l y,c0 −1)×L 1 values of a B j,q y,1 (s ji c0 , k q y,c0 , h y,c0 , l y,c0 ) B-Spline function for input C0 chroma pixel values s ji y , for i=0, 1, . . . , L 1 , at uniformly distributed knot points k p y,c0 at distance h y,c0 , order l y,c0 , and T y,c0 total k p y,c0 knot points; and
A y,c1 (s ji c1 , k q y,c1 ) denotes (T y,c1 +l y,c1 −1)×L 2 values of a B j,r y,2 (s ji c1 , k r y,c1 , h y,c1 , l y,c1 ) B-Spline function for input C1 chroma pixel values s ji c1 , for i=0, 1, . . . , L 2 , at uniformly distributed knot points k p y,c1 at distance h y,c1 , order l y,c1 , and T y,c1 total k p y,c1 knot points.
6 . The method of claim 5 , wherein generating luma output values in the 3D-LUT further comprises computing
B j,t TPB,y ( s ji y ,s ji c0 ,s ji c1 ,k p y,y ,k q y,c0 ,h y,y ,h y,c0 ,h y,c1 ,l y,y ,l y,c0 ,l y,c1 )= A y,y ( s ji y ,k p y,y )· A y,c0 ( s ji c0 ,k q y,c0 )· A y,c1 ( s ji c1 ,k q y,c1 ); and
predicting the output values {circumflex over (v)} ji y in the output dynamic range as
v
ˆ
j
i
y
=
Σ
t
=
0
D
y
-
1
m
j
,
t
TPB
,
y
·
B
j
,
t
TPB
,
y
(
s
j
i
y
,
s
j
i
c
0
,
s
j
i
c
1
,
k
p
y
,
y
,
k
q
y
,
c
0
,
k
r
y
,
c
1
,
h
y
,
y
,
h
y
,
c
0
,
h
y
,
c
1
,
l
y
,
y
,
l
y
,
c
0
,
l
y
,
c
1
)
,
wherein
D y =(T y,y +l y,y −1)×(T y,c0 +l y,c0 −1)×(T y,c1 +l y,c1 −1), and m j,t TPB,y denote the prediction coefficients.
7 . The method of claim 5 , wherein computing output points for a B-spline function of the form B t (x, k t , h, l), wherein x denotes input pixel values, k t denotes uniformly distributed knot points with distance h, and l denotes its order,
B
t
(
x
,
k
t
,
h
,
l
)
=
{
B
t
1
(
x
,
k
t
,
h
)
for
l
=
1
B
t
2
(
x
,
k
t
,
h
)
for
l
=
2
B
t
3
(
x
,
k
t
,
h
)
for
l
=
3
,
wherein for an l-th order truncated polynomial defined as
(
x
-
a
)
+
l
=
{
(
x
-
a
)
l
x
≥
a
0
x
<
a
B
t
1
(
x
,
k
t
,
h
)
=
(
1
h
)
[
(
x
-
k
t
-
1
)
+
-
2
(
x
-
k
t
)
+
+
(
x
-
k
t
+
1
)
+
]
,
B
t
2
(
x
,
k
t
,
h
)
=
1
2
(
1
h
)
2
[
(
x
-
k
t
-
1
)
+
2
-
3
(
x
-
k
t
)
+
2
+
3
(
x
-
k
t
+
1
)
+
2
-
(
x
-
k
t
+
2
)
+
2
]
,
and
B
t
3
(
x
,
k
t
,
h
)
=
1
6
(
1
h
)
3
[
(
x
-
k
t
-
2
)
+
3
-
4
(
x
-
k
t
-
1
)
+
3
+
6
(
x
-
k
t
)
+
3
-
4
(
x
-
k
t
+
1
)
+
3
+
(
x
-
k
t
+
2
)
+
3
]
.
8 . The method of claim 7 , wherein B t (x, k t , h, l) is computed for Nx values that are uniformly distributed within the input dynamic range.
9 . The method of claim 7 , wherein B t (x, k t , h, l) is computed for Nx values that are non-uniformly distributed within the input dynamic range.
10 . A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with claim 1 .
11 . An apparatus comprising a processor and configured to perform the methods recited in claim 1 .