Systems and methods for semantically classifying and extracting shots in video
View Patent ↗The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.
1. A method for classifying videos based on video content, comprising the steps of:
receiving a video file, the video file including a plurality of frames, where each frame includes a plurality of pixels;
extracting a set of frames from the video file;
for each frame in the extracted set of frames, determining whether the frame comprises a poor classification frame, removing the one or more poor classification frames from the extracted set of frames;
dividing each frame in the extracted set of frames into one or more segments, where each segment includes relatively uniform image content;
extracting image features from each segment to form a feature vector associated with each segment;
generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, where each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories; and
for each pixel of each frame in the extracted set of frames, assigning an average of material classification score vectors associated with segments that include the pixel.
2. The method of claim 1 , further comprising the step of storing the material classification score vectors assigned to each pixel in a database for subsequent use in video file classifications.
3. The method of claim 1 , further comprising the step of combining adjacent segments based on similar image content features extracted from the segments.
4. The method of claim 1 , where poor classification frames are determined via one or more classifiers.
5. The method of claim 1 , where a poor classification frame comprises a frame associated with at least one of the following frame types:
a frame shot in low light,
a frame shot at night,
a blurry frame, and
and an undetermined frame.
6. The method of claim 1 , where image features comprise one or more features selected from the group comprising: color features, edge features, texture features, and shape features.
7. The method of claim 1 , where image features comprise data associated with image content.
8. The method of claim 1 , where the one or more material classifiers are hierarchically related.
9. The method of claim 1 , where the one or more predefined material content categories are selected from the group comprising: building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, now/ice, trees/plants, vehicle, water, and miscellaneous.
10. The method of claim 1 , where each of the one or more material classification scores represents the probability that a frame includes content associated with each of the respective predefined material content categories.
11. The method of claim 1 , where the video file comprises a shot of video.
12. A non-transitory computer readable medium storing instructions, which when executed by one or more processors, cause performance of:
receiving a video file, the video file including a plurality of frames, where each frame includes a plurality of pixels;
extracting a set of frames from the video file;
for each frame in the extracted set of frames, determining whether the frame comprises a poor classification frame, removing the one or more poor classification frames from the extracted set of frames;
dividing each frame in the extracted set of frames into one or more segments, where each segment includes relatively uniform image content;
extracting image features from each segment to form a feature vector associated with each segment;
generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, where each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories; and
for each pixel of each frame in the extracted set of frames, assigning an average of material classification score vectors associated with segments that include the pixel.
13. The non-transitory computer readable medium as recited in claim 12 , further comprising the step of storing the material classification score vectors assigned to each pixel in a database for subsequent use in video file classifications.
14. The non-transitory computer readable medium as recited in claim 12 , further comprising the step of combining adjacent segments based on similar image content features extracted from the segments.
15. The non-transitory computer readable medium as recited in claim 12 , where poor classification frames are determined via one or more classifiers.
16. The non-transitory computer readable medium as recited in claim 12 , where a poor classification frame comprises a frame associated with at least one of the following frame types:
a frame shot in low light,
a frame shot at night,
a blurry frame, and
and an undetermined frame.
17. The non-transitory computer readable medium as recited in claim 12 , where image features comprise one or more features selected from the group comprising: color features, edge features, texture features, and shape features.
18. The non-transitory computer readable medium as recited in claim 12 , where the one or more predefined material content categories are selected from the group comprising: building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.
19. The non-transitory computer readable medium as recited in claim 12 , where each of the one or more material classification scores represents the probability that a frame includes content associated with each of the respective predefined material content categories.
20. An apparatus comprising:
a subsystem, implemented at least partially in hardware, that receives a video file, the video file including a plurality of frames, where each frame includes a plurality of pixels;
a subsystem, implemented at least partially in hardware, that extracts a set of frames from the video file;
a subsystem, implemented at least partially in hardware, that, for each frame in the extracted set of frames, determines whether the frame comprises a poor classification frame, removing the one or more poor classification frames from the extracted set of frames;
a subsystem, implemented at least partially in hardware, that divides each frame in the extracted set of frames into one or more segments, where each segment includes relatively uniform image content;
a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;
a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, where each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories; and
a subsystem, implemented at least partially in hardware, that, for each pixel of each frame in the extracted set of frames, assigns an average of material classification score vectors associated with segments that include the pixel.