Learning-based automatic commercial content detection
View Patent ↗Systems and methods for learning-based automatic commercial content detection are described. In one aspect, program data is divided into multiple segments. The segments are analyzed to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content. The context-based features are a function of single-side left and/or right neighborhoods of segments of the multiple segments.
1. A method for learning-based automatic commercial content detection, the method comprising:
dividing program data into multiple segments;
analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and
wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments; and
wherein the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C i , and wherein the method further comprises:
calculating each of the (2n+1) neighborhoods as follows:
N
k
=
[
N
s
k
,
N
e
k
]
=
{
[
min
(
e
j
+
α
k
,
0
)
,
e
i
]
[
s
i
,
e
i
]
[
s
i
,
min
(
s
i
+
α
k
,
L
)
]
k
<
0
k
=
0
k
>
0
,
wherein N k represents 2n+1, n representing a number of neighborhoods left and/or right of C i , [s i , e i ] denoting start and end frame numbers for C i and start and end times for C i , N k s represents a start frame number for N k , N k e represents an end frame number for N k , L indicating a length of the program data, kεZ,|k|≦n, and α a comprising a time step.
2. The method of claim 1 , wherein the visual feature set comprises segment-based features and time-based features.
3. The method of claim 1 , wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.
4. The method as of claim 1 , wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequency.
5. The method of claim 1 , wherein the method further comprises classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.
6. The method of claim 5 , wherein classifying further comprises:
training a Support Vector machine (SVM) classification model with commercial content; and
wherein the trained commercial content classification model comprises the SVM classification model.
7. The method of claim 1 , wherein the method further comprises:
classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and
post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.
8. The method of claim 7 , wherein post-processing further comprises:
grouping segments into scenes; and
merging scenes into the commercial and/or non-commercial blocks.
9. The method of claim 8 , wherein post-processing further comprises evaluating features of the commercial and/or non-commercial blocks to remove short scenes, remove embedded non-commercial content from long commercial scenes, detect long commercial portions in a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes and/or segments.
10. The method of claim 8 , wherein post-processing further comprises: responsive to applying the one or more heuristic criteria to remove shod scenes, remove embedded non-commercial content from long commercial scenes, detect long commercial portions in a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.
11. A method for learning-based automatic commercial content detection, the method comprising:
dividing program data into multiple segments;
analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content, the context-based features being a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments, the analyzing comprising further:
a calculating the context-based feature sets from segment-based visual features as an average value of visual features of S k , S k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S k ={C j k :0≦j<M k }={C i :C i ∩N k ≠Φ}, M k being a number of segments in S k , and wherein N k represents 2n+1 neighborhoods, n represents a number of neighborhoods left and/or right of a current segment C i , S k is a set of segments that are partially or totally included in N k , C k j represents is a j-th element of S k , M k represents a total number of elements in S k , and Φ represents an empty set.
12. A computer-readable medium for learning-based automatic commercial content detection, the computer-readable medium comprising computer-program executable instructions executable by a processor for:
dividing program data into multiple segments;
analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content, the context-based features being a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments; and
calculating the context-based feature sets from segment-based visual features as an average value of visual features of S k , S k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S k ={C j k :0≦j<M k }={C i :C i ∩N k ≠Φ}, M k being a number of segments in S k , and
wherein N k represents 2n+1 neighborhoods, n represents a number of neighborhoods left and/or right of a current segment C i , S k is a set of segments that are partially or totally included in N k , C k j represents is a j-th element of S k , M k represents a total number of elements in S k , and Φ represents an empty set.
13. The computer-readable medium of claim 12 , wherein the visual feature set comprises segment-based features and time-based features.
14. The computer-readable medium of claim 12 , wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.
15. The computer-readable medium of claim 12 , wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequency.
16. The computer-readable medium of claim 12 , wherein the computer-program instructions further comprise instructions for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.
17. The computer-readable medium of claim 16 , wherein the computer-program instructions for classifying further comprise instructions for:
training a Support Vector machine (SVM) classification model with commercial content; and
wherein the trained commercial content classification model comprises the SVM classification model.
18. The computer-readable medium of claim 12 , wherein the computer-program instructions further comprise instructions for:
classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and
post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.
19. The computer-readable medium of claim 18 , wherein the computer-program instructions for post-processing further comprise instructions for:
grouping segments into scenes; and
merging scenes into the commercial and/or non-commercial blocks.
20. The computer-readable medium of claim 19 , wherein post-processing farther comprises:
determining whether to apply one or more heuristic criteria to remove short scenes, double check long commercial scenes, detect long commercial portions of a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes; and
responsive to applying the one or more heuristic criteria, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.
21. A computer-readable medium for learning-based automatic commercial content detection, the computer-readable medium comprising computer-program executable instructions executable by a processor for:
dividing program data into multiple segments;
analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content the context-based features being a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments, the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C; and
calculating each of the (2n+1) neighborhoods as follows:
N
k
=
[
N
s
k
,
N
e
k
]
=
{
[
min
(
e
j
+
α
k
,
0
)
,
e
i
]
[
s
i
,
e
i
]
[
s
i
,
min
(
s
i
+
α
k
,
L
)
]
k
<
0
k
=
0
k
>
0
,
wherein N k represents 2n+1, n representing a number of neighborhoods left and/or right of C i , [s i , e i ] denoting start and end frame numbers for C i and start and end times for C i , N k s represents a start frame number for N k , N k e represents an end frame number for N k , L indicating a length of the program data, kεZ,|k|≦n, and α comprising a time step.
22. A computing device for learning-based automatic commercial content detection, the computing device comprising:
a processor; and
a memory coupled to the processor, the memory comprising computer-program executable instructions executable by the processor for:
dividing program data into multiple segments;
analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content, the context-based features being a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments; and
wherein the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C i , and wherein the computer-program instructions farther comprise instructions for:
calculating each of the (2n+1) neighborhoods as follows:
N
k
=
[
N
s
k
,
N
e
k
]
=
{
[
min
(
e
j
+
α
k
,
0
)
,
e
i
]
[
s
i
,
e
i
]
[
s
i
,
min
(
s
i
+
α
k
,
L
)
]
k
<
0
k
=
0
k
>
0
,
wherein N k represents 2n+1, n representing a number of neighborhoods left and/or right of C i , [s i , e i ] denoting start and end frame numbers for C i and start and end times for C i , N k s represents a start frame number for N k , N k e represents an end frame number for N k , L indicating a length of the program data, kεZ,|k|≦n, and α comprising a time step.
23. The computing device of claim 22 , wherein the visual feature set comprises segment-based features and time-based features.
24. The computing device of claim 22 , wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.
25. The computing device of claim 22 , wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound, and audio break frequency.
26. The computing device of claim 22 , wherein the computer-program instructions further comprise instructions for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.
27. The computing device of claim 26 , wherein the computer-program instructions for classifying further comprise instructions for:
training a Support Vector machine (SVM) classification model with commercial content; and
wherein the trained commercial content classification model comprises the SVM classification model.
28. The computing device of claim 22 , wherein the computer-program instructions further comprise instructions for:
classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and
post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.
29. The computing device of claim 28 , wherein the computer-program instructions for post-processing further comprise instructions for:
grouping segments into scenes; and
merging scenes into the commercial and/or non-commercial blocks.
30. The computing device of claim 29 , wherein post-processing further comprises:
determining whether to apply one or more heuristic criteria to remove short scenes, double check tong commercial scenes, detect long commercial portions of a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes; and
responsive to applying the one or more heuristic criteria, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.
31. A computing device for learning-based automatic commercial content detection, the computing device comprising:
a processor; and
a memory coupled to the processor, the memory comprising computer-program executable instructions executable by the processor for:
dividing program data into multiple segments;
analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content, the context-based features being a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments, the analyzing comprising:
calculating context-based feature sets from segment-based visual features as an average value of visual features of S k , S k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S k ={C j k :0≦j<M k }={C i :C i ∩N k ≠Φ}, M k being a number of segments in S k , and
wherein N k represents 2n+1 neighborhoods, n represents a number of neighborhoods left and/or right of a current segment C i , S k is a set of segments that are partially or totally included in N k , C k j represents is a j-th element of S k , M k represents a total number of elements in S k , and Φ represents an empty set.
32. A computing device for learning-based automatic commercial content detection, the computing device comprising:
means for dividing program data into multiple segments;
means for analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and
wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments, the single-side left and/or right neighborhoods being (2n+1) neighborhoods of a current segment C; and
wherein the computing device further comprises:
means for calculating each of the (2n+1) neighborhoods as follows:
N
k
=
[
N
s
k
,
N
e
k
]
=
{
[
min
(
e
j
+
α
k
,
0
)
,
e
i
]
[
s
i
,
e
i
]
[
s
i
,
min
(
s
i
+
α
k
,
L
)
]
k
<
0
k
=
0
k
>
0
,
wherein n represents a number of neighborhoods left and/or right of C i , [s i , e i ] denoting start and end frame numbers for C i and start and end times for C i , N k s denotes a staff frame number of N k , N k e denotes an end frame number of N k , L indicates a length of the program data, kεZ,|k|≦n, and α comprises a time step.
33. The computing device of claim 32 , wherein the computing device is a set-top box, a digital video recorder, or a general purpose computing device.
34. The computing device of claim 32 , wherein the visual feature set comprises segment-based features and time-based features.
35. The computing device of claim 32 , wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.
36. The computing device of claim 32 , wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequency.
37. The computing device of claim 32 , wherein the computing device further comprises means for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.
38. A computing device as recited in claim 37 , wherein the means for classifying further comprise:
means for training a Support Vector machine (SVM) classification model with commercial content; and
wherein the trained commercial content classification model comprises the SVM classification model.
39. The computing device of claim 32 , wherein the computing device further comprises:
means for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and
means for post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.
40. The computing device of claim 39 , wherein the means for post-processing further comprise:
means for grouping segments into scenes; and
means for merging scenes into the commercial and/or non-commercial blocks.
41. The computing device of claim 39 , wherein the means for post-processing further comprises:
means for determining whether to apply one or more heuristic criteria to remove short scenes, double check long commercial scenes, detect long commercial portions of a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes; and
responsive to applying the one or more heuristic criteria, means for re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.
42. A computing device for learning-based automatic commercial content detection, the computing device comprising:
means for dividing program data into multiple segments;
means for analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content, the context-based features being a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments;
means for calculating the context-based feature sets from segment-based visual features as an average value of visual features of S k , S k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S k ={C j k :0≦j<M k }={C i :C i ∩N k ≠Φ}, M k being a number of segments in S k , and
wherein N k represents 2n+1 neighborhoods, n represents a number of neighborhoods left and/or right of a current segment C i , S k is a set of segments that are partially or totally included in N k , C k j represents is a j-th element of S k , M k represents a total number of elements in S k , and Φ represents an empty set.