Video concept classification using audio-visual atoms

1. A method for determining a classification for a video segment, comprising using a processor to perform the steps of:

a) breaking the video segment into a plurality of short-term video slices, each including a plurality of video frames and an audio signal;

b) analyzing the video frames for each short-term video slice to form a plurality of region tracks, wherein the region tracks provide an indication of the position of identified image regions in a plurality of video frames;

c) analyzing each region track to form a corresponding visual feature vector providing an indication of visual features for the image region, and a motion feature vector providing an indication of inter-frame motion for the image region;

d) analyzing the audio signal for each short-term video slice to determine an audio feature vector providing a characterization of the audio signal;

e) forming a plurality of short-term audio-visual atoms for each short-term video slice by combining the visual feature vector and the motion feature vector for a particular region track with the corresponding audio feature vector; and

f) using a classifier to determine a classification for the video segment responsive to the short-term audio-visual atoms.

2. The method of claim 1 wherein the short-term video slices correspond to a specified time interval, and wherein the specified time interval is less than or equal to the duration of the video segment.

3. The method of claim 1 wherein step b) includes:

i) extracting a plurality of point tracks from the short-term video slice;

ii) extracting a plurality of regions from the short-term video slice; and

iii) combining the extracted point tracks and the extracted regions to form a plurality of region tracks.

4. The method of claim 1 wherein the visual feature vector includes color moment elements, texture feature descriptor elements or edge direction histogram elements.

5. The method of claim 1 wherein the motion feature vector includes elements providing a representation of the speed and direction of motion for the region track.

6. The method of claim 1 wherein the audio feature vector includes elements providing a time-frequency representation of the energy in the audio signal.

7. The method of claim 1 wherein step f) includes computing weighted norm-2 distances between the short-term audio-visual atoms and a plurality of prototype audio-visual atoms to determine a closest prototype audio-visual atom.

8. The method of claim 1 wherein the classifier is a support vector machine classifier.

9. The method of claim 1 wherein the classifier is trained by determining short-term audio-visual atoms for a training set of video segments, where the training set of video segments have known classifications.

10. The method of claim 1 wherein the classifier is trained using a multiple instance learning algorithm.

11. A method for determining a representation of a video segment, comprising using a processor to perform the steps of:

a) breaking the video segment into a plurality of short-term video slices, each including a plurality of video frames and an audio signal;

b) analyzing the video frames for each short-term video slice to form a plurality of region tracks, wherein the region tracks provide an indication of the inter-frame motion for image regions that occur in a plurality of video frames;

c) analyzing each region track to form a corresponding visual feature vector and a motion feature vector;

d) analyzing the audio signal for each short-term video slice to determine an audio feature vector;

e) forming a plurality of short-term audio-visual atoms for each short-term video slice by combining the visual feature vector and the motion feature vector for a particular region track with the corresponding audio feature vector; and

f) combining the short-term audio-visual atoms to form a representation of the video segment.

12. A system comprising:

a data processing system; and

a memory system communicatively connected to the data processing system and storing instructions configured to cause the data processing system to implement a method for determining a classification for a video segment, wherein the instructions comprise:

a) breaking the video segment into a plurality of short-term video slices, each including a plurality of video frames and an audio signal;

b) analyzing the video frames for each short-term video slice to form a plurality of region tracks, wherein the region tracks provide an indication of the position of identified image regions in a plurality of video frames;

c) analyzing each region track to form a corresponding visual feature vector providing an indication of visual features for the image region, and a motion feature vector providing an indication of inter-frame motion for the image region;

d) analyzing the audio signal for each short-term video slice to determine an audio feature vector providing a characterization of the audio signal;

e) forming a plurality of short-term audio-visual atoms for each short-term video slice by combining the visual feature vector and the motion feature vector for a particular region track with the corresponding audio feature vector; and

f) using a classifier to determine a classification for the video segment responsive to the short-term audio-visual atoms.