System and method for generating localized contextual video annotation

1. A method for localized contextual video annotation, comprising:

segmenting a piece of video content into a plurality of segments based on a video segmentation unit;

parsing a respective segment generated from the piece of video content;

determining, based on the parsing, multiple input modalities for the segment, wherein a respective input modality indicates a form of content in the segment;

classifying the segment into a set of semantic classes by applying respective classifiers on the input modalities of the segment; and

determining an annotation corresponding to the set of semantic classes for the segment, wherein the annotation indicates semantic contextual information associated with the segment.

2. The method of claim 1 , wherein classifying the segment further comprises:

applying a corresponding classifier of the classifiers on a respective input modality for determining classification for the input modality;

determining a unified classification for the segment based on the respective classifications for the multiple input modalities; and

determining the annotation for the segment based on the unified classification.

3. The method of claim 2 , wherein determining the unified classification further comprises fusing the respective classifications for the multiple input modalities with each other to generate the unified classification.

4. The method of claim 1 , wherein the multiple input modalities include video frames separated from an audio signal of the segment; and

wherein classifying the segment further comprises applying a deep visual classifier of the classifiers on the video frames to generate visual classification for the segment.

5. The method of claim 1 , wherein the multiple input modalities include audio signal separated from video frames of the segment;

wherein classifying the segment further comprises:

decomposing the audio signal into a background signal and a speech signal;

applying an audio classifier of the classifiers on the background signal to generate background audio classification for the segment; and

applying an emotion classifier of the classifiers on the speech signal to generate emotion classification for the segment.

6. The method of claim 1 , wherein the multiple input modalities include textual information separated from audio-visual signals of the segment; and

wherein classifying the segment further comprises:

obtaining speech text representing verbal speeches in the segment;

aligning the textual information with the speech text; and

applying a text-based classifier of the classifiers on the aligned text to generate textual classification for the segment.

7. The method of claim 6 , further comprising obtaining a script of the movie and further aligning the script with the speech text.

8. The method of claim 1 , wherein classifying the segment further comprises:

obtaining respective feature embeddings from the classifiers for the multiple input modalities;

combining the feature embeddings to generate a unified embedding; and

applying a semantic classifier on the unified embedding to determine the unified classification.

9. The method of claim 8 , wherein combining the feature embeddings further comprises applying feature concatenation on the feature embeddings.

10. The method of claim 1 , wherein the annotation includes a set of keys indicating the semantic contextual information of the segment, wherein a respective key includes a value and a strength, wherein the value indicates a feature of the segment, and wherein the strength indicates the likelihood of the value being associated with the segment.

11. The method of claim 1 , wherein the video segmentation unit is one or more of: an act, a scene, a beat, and a shot of the piece of video content.

12. The method of claim 1 , wherein a respective semantic class corresponds to one of: action, danger, romance, friendship, and outdoor adventure.

13. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for localized contextual video annotation, the method comprising:

segmenting a piece of video content into a plurality of segments based on a video segmentation unit;

parsing a respective segment generated from the piece of video content;

generating, based on the parsing, multiple input modalities for the segment, wherein a respective input modality indicates a form of content in the segment;

classifying the segment into a set of semantic classes by applying respective artificial intelligence (AI) models on the input modalities of the segment; and

determining an annotation corresponding to the set of semantic classes for the segment, wherein the annotation indicates semantic contextual information associated with the segment.

14. The computer-readable storage medium of claim 13 , wherein classifying the segment further comprises:

applying a corresponding classifier of the classifiers on a respective input modality for determining classification for the input modality;

determining a unified classification for the segment based on the respective classifications for the multiple input modalities; and

determining the annotation for the segment based on the unified classification.

15. The computer-readable storage medium of claim 14 , wherein determining the unified classification further comprises fusing the respective classifications for the multiple input modalities with each other to generate the unified classification.

16. The computer-readable storage medium of claim 13 , wherein the multiple input modalities include video frames separated from an audio signal of the segment; and

wherein classifying the segment further comprises applying a deep visual classifier of the classifiers on the video frames to generate visual classification for the segment.

17. The computer-readable storage medium of claim 13 , wherein the multiple input modalities include audio signal separated from video frames of the segment;

wherein classifying the segment further comprises:

decomposing the audio signal into a background signal and a speech signal;

applying an audio classifier of the classifiers on the background signal to generate background audio classification for the segment; and

applying an emotion classifier of the classifiers on the speech signal to generate emotion classification for the segment.

18. The computer-readable storage medium of claim 13 , wherein the multiple input modalities include textual information separated from audio-visual signals of the segment; and

wherein classifying the segment further comprises:

obtaining speech text representing verbal speeches in the segment;

aligning the textual information with the speech text; and

applying a text-based classifier of the classifiers on the aligned text to generate textual classification for the segment.

19. The computer-readable storage medium of claim 18 , wherein the method further comprises obtaining a script of the movie and further aligning the script with the speech text.

20. The computer-readable storage medium of claim 13 , wherein classifying the segment further comprises:

obtaining respective feature embeddings from the classifiers for the multiple input modalities;

combining the feature embeddings to generate a unified embedding; and

applying a semantic classifier on the unified embedding to determine the unified classification.

21. The computer-readable storage medium of claim 20 , wherein combining the feature embeddings further comprises applying feature concatenation on the feature embeddings.

22. The computer-readable storage medium of claim 13 , wherein the annotation includes a set of keys indicating the semantic contextual information of the segment, wherein a respective key includes a value and a strength, wherein the value indicates a feature of the segment, and wherein the strength indicates the likelihood of the value being associated with the segment.

23. The computer-readable storage medium of claim 13 , wherein the video segmentation unit is one or more of: an act, a scene, a beat, and a shot of the piece of video content.

24. The computer-readable storage medium of claim 13 , wherein a respective semantic class corresponds to one of: action, danger, romance, friendship, and outdoor adventure.

25. A method for placing advertisement based on localized contextual video annotation, comprising:

segmenting a piece of video content into a plurality of segments based on a video segmentation unit;

parsing a respective segment generated from the piece of video content;

for determining, based on the parsing, multiple input modalities for the segment, wherein a respective input modality indicates a form of content in the segment;

classifying the segment into a set of semantic classes by applying respective artificial intelligence (AI) models on the input modalities of the segment;

determining an annotation corresponding to the set of semantic classes for the segment, wherein the annotation indicates semantic contextual information associated with the segment;

identifying an inter-segment availability (ISA) between segments of the piece of video content as a target location for placing an advertisement; and

sending annotations of a set of segments associated with the ISA to an advertisement system, wherein the set of segments includes one or more of: a preceding segment of the ISA and a subsequent segment of the ISA.

26. A method for facilitating discretionary viewing based on localized contextual video annotation, comprising:

segmenting a piece of video content into a plurality of segments based on a segmentation unit;

parsing a respective segment generated from the piece of video content;

determining, based on the parsing, multiple input modalities for the segment, wherein a respective input modality indicates a form of content in the segment;

classifying the segment into a set of semantic classes by applying respective classifiers on the input modalities of the segment;

determining an annotation corresponding to the set of semantic classes for the segment, wherein the annotation indicates semantic contextual information associated with the segment;

obtaining a viewing preference from a viewer of the video; and

determining a set of viewing segments from the plurality of segments based on the annotations for the plurality of segments, wherein the set of viewing segments complies with the viewing preference.