Content based video content segmentation
A method receives video content and metadata associated with video content. The method then extracts features of the video content based on the metadata. Portions of the visual, audio, and textual features are fused into composite features that include multiple features from the visual, audio, and textual features. A set of video segments of the video content is identified based on the composite features of the video content. Also, the segments may be identified based on a user query.
1. A method comprising:
in a video data analyzer of a first computing device, configuring an extraction, based on metadata associated with video content, of content features;
wherein the content features are selected from the group consisting of visual features of the video content, audio features of the video content, and textual features of the video content,
wherein one or more feature extractors corresponding to the content features are selected from the group consisting of a visual feature extractor for content features selected from visual features of the video content, an audio feature extractor for content features selected from audio features of the video content, and a text feature extractor for content features selected from textual features of the video content, and
wherein configuring the extraction comprises configuring the one or more selected feature extractors to extract the respective content features in accordance with one or more operating parameters that are used internally by the respective feature extractor, and that are tunable by the video data analyzer to alter an extraction behavior of the feature extractor based on the metadata;
creating a single data stream of fused information for rendering in a client computing device communicatively coupled to one or more distributed content servers, wherein the creating comprises:
fusing, in a plurality of fusion modules communicatively coupled to the one or more distributed content servers, portions of the content features into composite features that are generated from functions of the multiple features from the content features;
identifying, by one or more of the plurality of fusion modules, a plurality of video segments comprising one or more video segments of the video content based on the composite features; and
rendering the created single data stream, in a user interface of the client computing device, by rendering representations of the identified video segments.
2. The method of claim 1 , wherein some of the plurality of video segments are identified based on only one content feature.
3. The method of claim 1 , wherein identifying the plurality of video segments comprises combining non-contiguous segments from the video content into a segment.
4. The method of claim 1 , wherein the multiple features are based on at least two of the group consisting of visual features of the video content, audio features of the video content, and textual features of the video content.
5. The method of claim 1 , wherein:
the composite features include the multiple features from at least two of the visual feature extractor, the audio feature extractor, and the text feature extractor.
6. The method of claim 1 , wherein:
the extraction is performed by a plurality of extractors, and
the metadata is used to configure an extractor in the plurality of extractors to extract one of visual, audio, and textual features based on the metadata.
7. The method of claim 1 , wherein:
the identifying is performed by a plurality of fusion modules, and
the metadata is used to configure a fusion module in the plurality of fusion modules to fuse the multiple features into the composite features.
8. The method of claim 7 , wherein the fusion module determines a composite feature based on the metadata.
9. The method of claim 1 , further comprising classifying the plurality of video segments based on the metadata.
10. The method of claim 1 , wherein the metadata comprises program metadata received from an electronic program guide data source.
11. The method of claim 1 , further comprising:
displaying the plurality of video segments;
receiving a selection of one of the plurality of video segments; and
displaying the one of the plurality of video segments.
12. The method of claim 11 , further comprising adding supplemental content in association with the one of the plurality of video segments based on a feature associated with the one of the plurality of video segments.
13. The method of claim 12 , wherein the supplemental content is based on a type of user reaction to the one of the plurality of video segments.
14. An apparatus comprising:
a plurality of computer processors comprising a video data analyzer processor and one or more segment services processors;
at least one non-transitory computer readable storage memory coupled to each of the plurality of computer processors and comprising instructions that when executed by one or more of the computer processors cause the one or more of the computer processors to be configured for:
in the video data analyzer processor, configuring an extraction, based on metadata associated with video content, of content features;
wherein the content features are selected from the group consisting of visual features of the video content, audio features of the video content, and textual features of the video content,
wherein one or more feature extractors corresponding to the content features are selected from the group consisting of a visual feature extractor for content features selected from visual features of the video content, an audio feature extractor for content features selected from audio features of the video content, and a text feature extractor for content features selected from textual features of the video content, and
wherein configuring the extraction comprises configuring the one or more selected feature extractors to extract the respective content features in accordance with one or more operating parameters that are used internally by the respective feature extractor, and that are tunable by the video data analyzer to alter an extraction behavior of the feature extractor based on the metadata;
creating a single data stream of fused information for rendering in a client computing device communicatively coupled to one or more distributed content servers, wherein the creating comprises:
in a plurality of fusion modules in the segment services processors, fusing portions of the content features into composite features that include are generated from functions of the multiple features from the content features, wherein the segment services processors are communicatively coupled to the one or more distributed content servers;
identifying, by one or more of the plurality of fusion modules, a plurality of video segments comprising one or more video segments of the video content based on the composite features; and
rendering the created single data stream, in a user interface of the client computing device, by rendering representations of the identified video segments.
15. A method for creating a single data stream of fused information for rendering in a client computing device communicatively coupled to one or more distributed content servers, the method comprising:
receiving a search query comprising at least one word;
receiving a textual program index associated with each video program from a plurality of video programs stored on a content server;
identifying, by one or more of a plurality of fusion modules, matching video programs from the plurality of video programs based on the textual program index associated with each video program and the at least one word;
receiving a user selection of a matching video program from the matching video programs to identify a selected video program;
receiving a plurality of text records associated with the selected video program;
searching, by one or more of the plurality of fusion modules, the text records of the selected video program to identify matching text records based on the at least one word;
segmenting, by one or more of the plurality of fusion modules, at least one matching video program into a plurality of video segments that include at least one of the matching text records; and
rendering the created single data stream, in a user interface of the client computing device, by rendering representations of the identified video segments;
wherein the plurality of fusion modules is implemented by one or more segment services processors each comprising one or more computer processors, the one or more segment services processors communicatively coupled to the one or more distributed content servers.
16. The method of claim 15 , further comprising:
presenting a representation of the at least one matching video programs to a user; and
receiving a user selection that identifies a user-selected matching video program.
17. The method of claim 15 , further comprising:
presenting at least a portion of the generated at least one matching segment to a user.
18. The method of claim 15 , further comprising:
calculating a windowed text record rank from the plurality of text records associated with a matching video program;
identifying contiguous blocks of matching text records as a segment;
assigning a segment score to the segment based upon the text record rank of the segment; and
presenting at least one segment to a user based on the segment score associated with the at least one segment.
19. The method of claim 15 , wherein receiving the textual program index associated with each video program comprises:
creating a plurality of text records for each video program, wherein each text record comprises at least a start time and a representation of text for the text record;
creating the textual program index from the plurality of text records for each video program; and
storing the textual program index for each video program with an identifier for the video program.
20. The method of claim 15 , wherein the plurality of video segments are combined with previously generated segments based on extraction of features in the video program.