Scene-change detection using video stream pairs
A scene change is determined using a first and a second video signal, each representing the same scene or content, but at a different color grade (such as dynamic range). A set of prediction coefficients is generated to generate prediction signals approximating the first signal based on the second signal and a prediction model. A set of prediction error signals is generated based on the prediction signals and the first signal. Then, a scene change is detected based on the characteristics of the prediction error signals. Alternatively, a set of entropy values of the difference signals between the first and second video signals are computed, and a scene change is detected based on the characteristics of the entropy values.
1. A method for scene change detection, the method comprising:
accessing a first video signal and a second video signal, wherein both the first video signal and the second video signal represent the same video content but at a different color grade;
for a sequence of consecutive frames in the first and second video signals:
computing prediction coefficients for a prediction model to generate predicted frames of the first video signal given data of the second video signal;
computing prediction errors signals between the generated predicted frames and the corresponding frames of the first input signal; and
determining a scene change in the sequence of the consecutive frames based on the prediction error signals, wherein determining a scene change between frames j−1 and j in the sequence of the consecutive frames based on the prediction error signals further comprises computing whether
D j−1,j >D j,j+i
for i=1, 2, . . . , Ts, where Ts is an integer and D j,k denotes a prediction error computed for the k-th frame in the first video signal using prediction model coefficients computed with data of the first and second video signals up to the j-th frame.
2. The method of claim 1 , wherein a different color grade comprises a difference in the dynamic range between the first and second video signals.
3. The method of claim 2 , wherein the first video signal has a higher dynamic range than the second video signal.
4. The method of claim 1 , wherein determining a scene change in the sequence of the consecutive frames based on the prediction error signals comprises applying a one-dimensional or a two-dimensional edge detection algorithm to the prediction error signals.
5. The method of claim 1 , wherein determining a scene change in the sequence of the consecutive frames based on the prediction error signals further comprises:
computing a first set of prediction coefficients for the prediction model using data of the first and second video signal up-to a first frame (j−1);
applying the first set of prediction coefficients to the prediction model to generate predicted frames for the first frame and a sequence of frames before and after the first frame;
generating prediction errors between the generated predicted frames and the corresponding frames in the first video signal; and
determining a scene change after the first frame if an approximation of a first derivative of samples of the generated prediction errors is larger than a first threshold.
6. The method of claim 5 , wherein determining a scene change between frames j−1 (the first frame) and j comprises computing whether
θ
1
=
M
L
-
M
R
min
{
M
L
,
M
R
}
>
T
ave
is true, where T ave denotes the first threshold,
M
L
=
1
a
∑
i
=
-
a
-
1
D
j
-
a
,
j
+
1
,
M
R
=
1
b
+
1
∑
i
=
0
b
D
j
-
1
,
j
+
i
,
where a denotes the total number of frames used before the first frame and b denotes the total number of frames used after the first frame, and D j,k denotes a prediction error computed for the k-th frame of the first video signal using prediction model coefficients computed with data of the first and second video signals up to frame j.
7. The method of claim 1 wherein determining a scene change between frames j−1 and j in the sequence of the consecutive frames based on the prediction error signals further comprises computing whether:
| D j−3,j−1 −D j−3,j |>max{| D j−3,j−2 −D j−3,j−1 |,|D j−3,j −D j−3,j+1 |}
&& | D j−2,j−1 |D j−2,j |>max{| D j−2,j−2 −D j−2,j−1 |,|D j−2,j −D j−2,j+1 |}
&& | D j−1,j−1 −D j−1,j |>max{| D j−1,j−2 −D j−1,j−1 |,|D j−1,j −D j−1,j+1 |}
&& | D j,j−1 −D j,j |>max{| D j,j−2 −D j,j−1 |,|D j,j −D j,j+1 |},
is true, where D j,k denotes a prediction error computed for the k-th frame in the first video signal using prediction model coefficients computed with data of the first and second video signals up to the j-th frame.
8. The method of claim 1 wherein determining a scene change between frames j−1 and j in the sequence of the consecutive frames based on the prediction error signals further comprises computing whether
ω j−1 <min{ω j−2 ,ω j−3 }
is true, where
ω j−3 =variance{ D j−3,j−2 ,D j−3,j−1 ,D j−3,j ,D j−3,j+1 }
ω j−2 =variance{ D j−2,j−1 ,D j−2,j ,D j−2,j+1 ,D j−2,j+2 }
ω j−1 =variance{ D j−1,j ,D j−1,j+1 ,D j−1,j+2 ,D j−1,j+3 },
and D j,k denotes a prediction error computed for the k-th frame in the first video signal using prediction model coefficients computed with data of the first and second video signals up to the j-th frame.
9. The method of claim 1 , wherein the computed prediction errors are translated to a logarithmic domain before being applied to determine the scene change.
10. The method of claim 1 , wherein computing the prediction coefficients for the prediction model comprises applying the prediction model across a sliding window of input frame data.
11. The method of claim 1 , wherein the first video signal and the second video signal comprise three or more color channels and the steps to determine the scene change further comprise:
making a preliminary determination whether there is a scene change in the sequence of the consecutive frames for each of the color channels of the first and second video signals according to the steps of claim 1 ; and
making a final determination whether there is a scene change if the same scene change is preliminary determined in the majority of the three or more color channels.
12. The method of claim 1 , further comprising:
accessing a third video signal representing the same video content as the first and second video signals but at a color grade different than the color grades of the first and second video signals;
making a first preliminary determination whether there is a scene change in the sequence of the consecutive frames between the first and second video signals according to the steps of claim 1 ;
making a second preliminary determination whether there is a scene change in the sequence of the consecutive frames between the first and third video signals according to the steps of claim 1 , wherein the second video signal is the third video signal; and
making a final determination whether there is a scene change if a scene change in the first preliminary determination matches a scene change in the second preliminary determination.
13. The method of claim 1 , further comprising:
accessing a third video signal representing the same video content as the first and second video signals but at a color grade different than the color grades of the first and second video signals;
making a first preliminary determination whether there is a scene change in the sequence of the consecutive frames between the first and second video signals according to the steps of claim 1 ;
making a second preliminary determination whether there is a scene change in the sequence of the consecutive frames between the first and third video signals according to the steps of claim 1 , wherein the second video signal is the third video signal;
making a third preliminary determination whether there is a scene change in the sequence of the consecutive frames between the second and third video signals according to the steps of claim 1 , wherein the first video signal is the third video signal; and
making a final determination whether there is a scene change if the same scene change is determined in the majority of the preliminary determination steps.
14. An apparatus comprising a processor and configured to perform the method recited in claim 1 .
15. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing a method with a processor in accordance with claim 1 .