Task specific audio classification for identifying video highlights
View Patent ↗A method classifies segments of a video using an audio signal of the video and a set of classes. Selected classes of the set are combined as a subset of important classes, the subset of important classes being important for a specific highlighting task, the remaining classes of the set are combined as a subset of other classes. The subset of important classes and classes are trained with training audio data to form a task specific classifier. Then, the audio signal can be classified using the task specific classifier as either important or other to identify highlights in the video corresponding to the specific highlighting task. The classified audio signal can be used to segment and summarize the video.
1. A method for classifying a video, comprising the steps of:
defining a set of classes for classifying an audio signal of a video;
combining selected classes of the set as a subset of important classes, the subset of important classes is important for a specific highlighting task;
combining the remaining classes of the set as a subset of other classes;
training jointly the subset of important classes and the subset of other classes with training audio data to form a task specific classifier;
classifying the audio signal using the task specific classifier as either important or other to identify highlights in the video corresponding to the specific highlighting task;
representing the subset of important classes with a first Gaussian mixture model; and
representing the subset of other classes with a second Gaussian mixture model, in which a number C of the subsets of classes is 2, and there are N train samples in a vector x of the training audio data, and each sample x i has an associated class label y i that takes on values 1 to C, and the task specific classifier has a form:
f
(
x
;
m
)
=
arg
max
y
p
(
x
|
y
,
m
y
,
Θ
y
)
,
where
arg
max
y
p
(
x
|
y
,
m
y
,
Θ
y
)
is a value of y for which p(x|y, m y , Θ y ) has a largest value, p stands for a condition probability, where the symbol | indicates a condition of the probability of the sample x given the class label y, m=[m 1 , . . . , m c ] T is a number of mixture components for each Gaussian mixture model, and Θ represents parameters of each Gaussian mixture model.
2. The method of claim 1 , further comprising:
segmenting the video according to the classified audio signal into important segments and other segments; and
combining the important segments into a summary of the video.
3. The method of claim 1 , further comprising:
partitioning the audio signal into frames;
extracting audio features from each frame;
classifying each frame according to the audio features as either an important frame or an other frame.
4. The method of claim 3 , in which the audio features are modified discrete cosine transforms.
5. The method of claim 1 , in which the video is of a sporting event, and the specific highlighting task is identifying highlights in the video, and the set of classes includes a mixture of excited speech and cheering, applause, cheering, normal speech, and music classes, and the subset of important classes includes the mixture of excited speech and cheering, and the subset of other classes includes applause, cheering, normal speech, and music.
6. The method of claim 1 , in which the training jointly uses K-fold cross validation.
7. The method of claim 1 , in which the training jointly optimizes an estimate of classification.
8. The method of claim 1 , in which the classifying assigns labels, and further comprising:
determining importance levels of the labels according to the specific highlighting task.
9. The method of claim 1 , in which the training audio data includes a validation set with N test samples, and associated labels (x i , y i ), and an empirical test error on the validation set for a particular m is:
TestErr
(
m
)
=
1
N
test
∑
i
=
1
N
test
1
-
δ
(
y
i
-
f
(
x
i
;
m
)
)
,
where TestErr(m) is the empirical test error, δ is a scalar wherein δ is 1 when y i =f(x i ; m), and 0 otherwise.
10. The method of claim 1 , in which an optimum number of mixture components {circumflex over (m)} is selected according to:
m
^
=
arg
min
m
TestErr
(
m
)
,
where TestErr(m) is an empirical test error, and where
arg
min
m
TestErr
(
m
)
is a value of m for which TestErr(m) has a smallest value.
11. A system for classifying a video, comprising:
a memory configured to store a set of classes for classifying an audio signal of a video;
means for combining selected classes of the set as a subset of important classes, the subset of important classes is important for a specific highlighting task;
means for combining the remaining classes of the set as a subset of other classes;
means for training jointly the subset of important classes and the subset of other classes with training audio data to form a task specific classifier;
means for classifying the audio signal using the task specific classifier as either important or other to identify highlights in the video corresponding to the specific highlighting task;
means for representing the subset of important classes with a first Gaussian mixture model; and
means for representing the subset of other classes with a second Gaussian mixture model, in which a number C of the subsets of classes is 2, and there are N train samples in a vector x of the train in audio data, and each sample x i has an associated class label y i that takes on values 1 to C, and the task specific classifier has a form:
f
(
x
;
m
)
=
arg
max
y
p
(
x
|
y
,
m
y
,
Θ
y
)
,
where
arg
max
y
p
(
x
|
y
,
m
y
,
Θ
y
)
is a value of y for which p(x|y, m y , Θ y ) has a largest value, p stands for a condition probability, where the symbol | indicates a condition of the probability of the sample x given the class label y, m=[m 1 , . . . , m c ] T is a number of mixture components for each Gaussian mixture model, and Θ represents parameters of each Gaussian mixture model.