Feature identification of events in multimedia
View Patent ↗A method detects events in multimedia. Features are extracted from the multimedia. The features are sampled using a sliding window to obtain samples. A context model is constructed for each sample. The context models form a time series. An affinity matrix is determined from the time series models and a commutative distance metric between each pair of context models. A second generalized eigenvector is determined for the affinity matrix, and the samples are then clustered into events according to the second generalized eigenvector.
1. A computer-implemented method for detecting events in multimedia, comprising the steps of:
extracting features from the multimedia;
sampling the features using a sliding window to obtain a plurality of samples;
constructing a context model for each sample;
determining an affinity matrix from the models and a commutative distance metric between each possible pair of context models;
determining a second generalized eigenvector for the affinity matrix;
clustering the plurality of samples into events according to the second generalized eigenvector; and
generating a summary of the multimedia according to the events.
2. The method of claim 1 , in which the features represent usual events of a background process and unusual events of a foreground process.
3. The method of claim 1 , further comprising:
labeling the features with labels.
4. The method of claim 3 , in which the labels are discrete.
5. The method of claim 3 , in which the labels are continuous.
6. The method of claim 1 , in which the multimedia includes an audio portion and audio features are selected from the group consisting of pitch, amplitude, Mel frequency cepstral coeficients (MFCC), Modified Discrete Cosine Transform (MDCT) coefficients from an ac-3 stream, speech, and music.
7. The method of claim 1 , in which the multimedia includes a video portion, and video features are selected multimedia from the group consisting of color, motion, texture, and MPEG-7 descriptors.
8. The method of claim 1 , in which the sliding window has a length W L , and a step size W S , and wherejn the length W L determines a reliability of the context model, and the step size W S determines a resolutlon of the events.
9. The method of claim 1 , in which the second generalized eigenvector ranks the events.
10. The method of claim 9 , fruther comprising:
summarizing the multimedia according to the ranked events.
11. The method of claim 2 , in which the background process is memoryless, and the context model is an unconditional probability distribution function.
12. The method of claim 2 , in which the background process has memory, and the context model is a hidden Markov model.
13. The method of claim 1 , in which the context model is a Gaussian mixture model.
14. The method of claim 1 , in which the sliding window has a length W L , and the commutative distance metric d for each pair of context models λ 1 and λ 2 constructed from samples O 1 and O 2 , respectively is
d
(
λ
1
,
λ
2
)
=
1
/
W
L
(
log
P
(
O
1
❘
λ
1
)
+
log
P
(
O
2
❘
λ
2
)
-
log
P
(
O
1
❘
λ
2
)
-
log
P
(
O
2
❘
λ
1
)
)
.
15. The method of claim 1 , in which the multimedia is a sports video and the events are significant events.
16. The method of claim 15 , in which the events include excited speech and cheering.
17. The method of claim 1 , in which the multimedia is a surveillance video and the events are unusual events.