Identifying scene boundaries using group sparsity analysis
A method for identifying a set of key video frames from a video sequence comprising extracting feature vectors for each video frame and applying a group sparsity algorithm to represent the feature vector for a particular video frame as a group sparse combination of the feature vectors for the other video frames. Weighting coefficients associated with the group sparse combination are analyzed to determine video frame clusters of temporally-contiguous, similar video frames. The video sequence is segmented into scenes by identifying scene boundaries based on the determined video frame clusters.
1. A method for determining scene boundaries within a video sequence including a time sequence of video frames, each video frame including an array of image pixels having pixel values, comprising:
a) selecting a set of video frames from the video sequence;
b) extracting a feature vector for each video frame in the set of video frames;
c) applying a group sparsity algorithm to represent the feature vector for a particular video frame as a group sparse combination of the feature vectors for the other video frames in the set of video frames, each feature vector for the other video frames in the group sparse combination having an associated weighting coefficient, wherein the weighting coefficients for feature vectors corresponding to other video frames that are most similar to the particular video frame are non-zero, and the weighting coefficients for feature vectors corresponding to other video frames that are most dissimilar from the particular video frame are zero;
d) analyzing the weighting coefficients to determine a video frame cluster of temporally-contiguous, similar video frames that includes the particular video frame;
e) repeating steps c)-d) for a plurality of particular video frames to provide a plurality of video frame clusters;
f) identifying one or more scene boundaries corresponding scenes in the video sequence based on the locations of boundaries between the determined video frame clusters; and
g) storing an indication of the identified scene boundaries in a processor-accessible memory;
wherein the method is performed, at least in part, using a data processor.
2. The method of claim 1 wherein the extraction of the feature vector for a video frame includes:
forming a frame vector including pixel values from the video frame;
defining a basis function set including a plurality of basis functions, each basis function being a vector having an identical length to the frame vector;
forming products between the frame vector and of the basis functions to determine corresponding feature coefficients;
forming the feature vector by collecting the feature coefficients for each basis function.
3. The method of claim 2 wherein the frame vector includes pixel values for a subset of the image pixels in the video frame selected according to a predefined sampling pattern.
4. The method of claim 2 wherein the basis functions are vectors of random numbers.
5. The method of claim 2 wherein the basis functions are Fourier transform basis functions, discrete cosine transform basis functions, or wavelet basis functions.
6. The method of claim 1 wherein the determined video frame cluster includes only the particular video frame and video frames that follow the particular video frame in the video sequence.
7. The method of claim 6 wherein the video frames included in the video frame cluster are determined by identifying the next video frame having an insignificant weighting coefficient that is smaller than a predefined threshold.
8. The method of claim 1 wherein the video frame cluster includes the temporally-contiguous video frames where the corresponding feature vectors have weighting coefficients that exceed a predefined threshold.
9. The method of claim 1 wherein similarity values are determined between adjacent video frame clusters, and wherein adjacent video frame clusters having similarity values that exceed a predefined threshold are merged to form video segments corresponding to scenes in the video sequence, and wherein boundaries of the video segments define the scene boundaries.
10. The method of claim 9 wherein the similarity values are determined by comparing representative video frames selected from within the video frame clusters.
11. The method of claim 9 wherein the similarity values are determined responsive to global color histograms determined for the video frame clusters.
12. The method of claim 9 wherein the similarity values are determined responsive to block-based color histograms determined for blocks of pixels in one or more video frames in the video frame clusters.
13. The method of claim 1 wherein indications of the determined scene boundaries are stored as metadata in association with a video file used to store the video sequence.
14. The method of claim 13 wherein the metadata is stored in the video.
15. The method of claim 1 wherein video frames within one or more scenes are extracted from the video sequence and stored in a separate video file.