Searching multi-media databases using multi-media queries
View Patent ↗A system and method for generating responsibility vectors associated with multi-media files (e.g., audio and/or video files) is provided. The responsibility vectors are based upon responsibility of mixture components fitted to a mixture model for frames of the files. The responsibility vectors can be grouped based upon clustering related to extracted identifiable features of frames of the multi-media files. Once generated, responsibility vectors can be searched by a multi-media searching system. Also provided is a system for multi-media searching based, at least in part upon responsibility vectors associated with a query segment and multi-media files. The system can generate a query profile based, at least in part, upon responsibility vectors of frames of the query segment. The system can further generate segment profiles of segments of the multi-media files. The system can then provide information associated with a likelihood that a particular file of the multi-media files includes the query segment based, at least in part, upon the query profile and a segment profile of the particular file.
1. A computer-implemented multi-media responsibility vector generation system, comprising:
a subband extractor that extracts subband signals from frames of files, the files comprising at least one of audio information and video information;
a feature extractor that forms a feature vector for at least some of the frames based, at least in part, upon features extracted from the subband signals;
a feature cluster component that clusters the files based, at least in part, upon vector quantization of the extracted features; and,
a responsibility vector generator that generates a responsibility vector for at least some of the frames based, at least in part, upon responsibility of mixture components fitted to a mixture model for the frame.
2. The system of claim 1 , at least one mixture component of the mixture model being Gaussian.
3. The system of claim 2 , the mixture model being based, at least in part, upon the following equation:
p
(
x
n
)
=
∑
s
=
1
S
p
(
x
n
|
s
n
=
s
)
p
(
s
n
=
s
)
where x n is data of an nth frame,
S is a quantity of components of the mixture model,
p(x n |s n =s) is a component of the mixture model, and,
p(s n =s) are mixing proportions.
4. The system of claim 3 , the responsibility vectors being based, at least in part, upon the following equation:
γ
sn
=
p
(
s
n
=
s
|
x
n
)
=
p
(
x
n
|
s
n
=
s
)
p
(
s
n
=
s
)
∑
s
′
p
(
x
n
|
s
n
=
s
′
)
p
(
s
n
=
s
′
)
where γ sn is the responsibility of component s of frame n.
5. The system of claim 1 , the extracted features including at least one of log-spectra, cepstra, temporal derivatives of log-spectra and temporal derivates of cepstra.
6. The system of claim 1 , further composing a multi-media data store that stores at least some of the files.
7. The system of claim 1 , further comprising a responsibility vector store that stores responsibility vectors associated with frames of at least some of the files.
8. A computer-implemented multi-media searching system, comprising:
a query component that calculates responsibility vectors for frames of a multi-media query segment, the multi-media query component providing a query profile based, at least in part, upon the responsibility vectors of the frames;
a segment component that retrieves responsibility vectors for segments of multi-media files, the segment component generating segment profiles of at least one segment of the multi-media files, a segment profile for a segment being based, at least in part, upon responsibility vectors of the segment; and,
a sort component that provides information associated with a likelihood that a particular multi-media file includes the multi-media query segment based, at least in part, upon the query profile and a segment profile of a segment of the particular multi-media file.
9. The system of claim 8 , the query profile being based upon an average of the responsibility vectors over the frames of the query segment.
10. The system of claim 8 , the segment profile being based upon an average of the responsibility vectors over the frames of the segment.
11. The system of claim 8 , the query component further extracting subband signals from frames of the query segment.
12. The system of claim 11 , calculation of the responsibility vectors being based, at least in part, upon responsibility of mixture components fitted to a mixture model for the frames of the query segment.
13. The system of claim 12 , at least one mixture component of the mixture model being Gaussian.
14. The system of claim 13 , the mixture model being based, at least in part, upon the following equation:
p
(
x
n
)
=
∑
s
=
1
S
p
(
x
n
|
s
n
=
s
)
p
(
s
n
=
s
)
where x n is data of an nth frame of the query segment,
S is a quantity of components of the mixture model,
p(x n |s n =s) is a component of the mixture model, and,
p(s n =s) are mixing proportions.
15. The system of claim 13 , the responsibility vectors of the query segment being based, at least in part, upon the following equation:
γ
sn
=
p
(
s
n
=
s
|
x
n
)
=
p
(
x
n
|
s
n
=
s
)
p
(
s
n
=
s
)
∑
s
′
p
(
x
n
|
s
n
=
s
′
)
p
(
s
n
=
s
′
)
where γ sn is the responsibility of component s of frame n.
16. The system of claim 8 , further comprising a display component that displays information associated with a likelihood that a particular file includes the query segment received from the sort component.
17. The system of claim 8 , further comprising a responsibility vector store that stores responsibility vectors associated with frames of at least some of the files.
18. The system of claim 8 , further comprising a multi-media data store that a multi-media data store that stores at least some of the files.
19. The system of claim 8 , the sorting component further calculating a Kullback-Leibler distance between the query profile and at least some of the segment profiles, the sorting component further using the Kullback-Leibler distances to sort the segment profiles based, at least in part, upon the segment profile's similarity to the query profile.
20. The system of claim 19 , calculation of the Kullback-Leibler distance being based, at least in part, upon the following equation:
D
(
r
,
q
)
=
∑
s
=
1
S
r
s
log
r
s
q
s
where r s is a particular segment profile,
q s is the query profile,
S is a quantity of segment profiles, and,
D(r,q)) is the Kullback-Leibler distance.
21. A computer-implemented method for storing responsibility vectors, comprising:
extracting subband signals from frames of segments of files;
extracting features from the extracted subband signals;
clustering files based, at least in part, upon vector quantization of the extracted features;
fitting a mixture model to the extracted subband signals, for each frame; and,
generating a responsibility vector for each frame based, at least in part, upon responsibility of mixture components of the mixture model.
22. A computer-implemented method for searching a multi-media database, comprising:
calculating responsibility vectors for frames of a multi-media query segment; generating a query profile based, at least in part, upon the responsibility vectors of the frames;
generating segment profiles of at least one segment of multi-media files, a segment profile for a segment being based, at least in part, upon responsibility vectors of the segment; and,
providing information associated with a likelihood that a particular multi-media file includes the multi-media query segment based, at least in part, upon the query profile and a segment profile of a segment of the particular multi-media file.
23. The method of claim 22 , calculating responsibility vectors for frames of the query segment comprising at least one of the following acts:
extracting subband signals from frames of the query segment; and,
fitting a mixture model to the extracted subband signals, for each frame.
24. The method of claim 22 , providing information further comprising:
calculating a Kullback-Leibler distance between the query profile and at least some of the segment profiles; and,
using the Kullback-Leibler distances to sort the segment profiles based, at least in part, upon the segment profile's similarity to the query profile.
25. A computer readable medium storing computer executable components of a multi-media responsibility vector generation system, comprising:
a subband extractor component that extracts subband signals from frames of files, the files comprising at least one of audio information and video information;
a feature extractor component that forms a feature vector for at least some of the frames based, at least in part, upon features extracted from the subband signals;
a feature cluster component that clusters the files based, at least in pan, upon vector quantization of the extracted features; and,
a responsibility vector generator component that generates a responsibility vector for at least some of the frames based, at least in part, upon responsibility of mixture components fitted to a mixture model for the frame.
26. A computer readable medium storing computer executable components of a multi-media searching system, comprising:
a query component that calculates responsibility vectors for frames of a multi-media query segment,the query component providing a query profile based, at least in part, upon the responsibility vectors of the frames;
a segment component that retrieves responsibility vectors for segments of multi-media files, the segment component generating segment profiles of at least one segment of the multi-media files, a segment profile for a segment being based, at least in part, upon responsibility vectors of the segment; and,
a sort component that provides information associated with a likelihood that a particular multi-media file includes the multi-media query segment based, at least in part, upon the query profile and a segment profile of a segment of the particular multi-media file.
27. A computer-implemented multi-media responsibility vector generation system, comprising:
means for extracting subband signals from frames of files, the files comprising at least one of audio information and video information; means for extracting a features and forming a feature vector for at least some of the frames based, at least in part, upon features extracted from the subband signals;
means for clustering the files based, at least in part, upon vector quantization of the extracted features; and, means for generating a responsibility vector for at least some of the frames based, at least in part, upon responsibility of mixture components fitted to a mixture model for the frame.
28. A computer-implemented multi-media searching system, comprising:
means for calculating responsibility vectors for frames of a query segment;
means for calculating a query profile based, at least in part, upon the responsibility vectors of the frames;
means for retrieving responsibility vectors for segments of multi-media files;
means for generating segment profiles of at least one segment of the multi-media files,
a segment profile for a segment being based, at least in part, upon responsibility vectors of the segment; and,
means for providing information associated with a likelihood that a particular multi-media file includes the query segment based, at least in part, upon the query profile and a segment profile of a segment of the particular multi-media file.