Topical based media content summarization system and method
Disclosed herein is an automated approach for summarizing media content using descriptive information associated with the media content. For example and without limitation, the descriptive information may comprise a title associated with the media content. One or more segments of the media content may be identified to form a media content summary based on each segment's respective similarity to the descriptive information, which respective similarity may be determined using a media content and auxiliary data feature spaces. A shared dictionary of canonical patterns generated using the media content and auxiliary data feature spaces may be used in determining a media content segment's similarity to the descriptive information.
1. A method comprising:
obtaining, by a digital content summarization system server and for a first media content item having associated descriptive information, a plurality of second media content items as auxiliary data, the obtaining using the descriptive information associated with the first media content item, the first media content item comprising a plurality of units;
generating, by the digital content summarization system server, a media content item feature space comprising a first number of feature vectors comprising feature descriptor values representing the first media content item, the generating further comprising generating an auxiliary data feature space comprising a second number of feature vectors comprising feature descriptor values representing the plurality of second media content items as the auxiliary data, obtained using the descriptive information;
identifying, by the digital content summarization system server, a plurality of segments of the first media content item, each segment comprising at least one unit of the first media content item's plurality of units;
scoring, by the digital content summarization system server, each segment of the plurality of segments of the first media content item, scoring a segment of the plurality of segments comprising determining a distance between the auxiliary data feature space and at least one feature vector of the first number of feature vectors, the at least one feature vector representing the segment, each segment's score representing a measure of similarity of the segment to the descriptive information represented by the auxiliary data feature space;
identifying, by the digital content summarization system server, at least one segment of the plurality of segments of the first media content item as more similar to the descriptive information, based on its determined distance to the auxiliary data feature space representing the descriptive information, relative to others of the plurality of segments using the scoring of the plurality of segments; and
generating, by the digital content summarization system server, a summary of the first media content item, the summary comprising the at least one segment, of the plurality of segments of the first media content item, identified as being more similar to the descriptive information.
2. The method of claim 1 , the descriptive information comprising a title of the first media content item.
3. The method of claim 1 , scoring further comprising:
determining, for each segment, a unit-level score for each of the at least one unit of the segment, determining the unit-level score for a unit comprising determining a distance between the auxiliary data feature space and a feature vector of the first number of feature vectors, the feature vector representing the unit, each unit level score representing a measure of similarity of a respective one of the at least one unit to the descriptive information represented by the auxiliary data feature space; and
scoring each segment using the unit-level score determined for each of the at least one unit of the segment.
4. The method of claim 3 , each segment's score comprising an average unit-level score determined using each unit-level score determined for each of the at least one unit of the segment.
5. The method of claim 3 , the first media content item is a video content item, each unit of the first media content item is a frame and each segment comprises at least one frame.
6. The method of claim 1 , generating a media content item feature space and an auxiliary data feature space further comprising:
generating, by the digital content summarization system server, the media content item feature space by generating, for each of the plurality of units of the first media content item, a plurality of feature descriptor values, each feature descriptor value corresponding to a feature of a set of features; and
generating, by the digital content summarization system server, the auxiliary data feature space by generating, for each second media content item of the plurality of second media content items as the auxiliary data, a plurality of feature vectors each comprising a plurality of feature descriptor values, each feature descriptor value of the plurality of feature descriptor values corresponding to a feature of the set of features used in generating the plurality of feature descriptor values for each unit of the second media content item.
7. The method of claim 1 , the scoring further comprising:
determining, by the digital content summarization system server and using the media content item feature space and the auxiliary data feature space, a shared dictionary comprising a plurality of canonical patterns shared by the first media content item and the auxiliary data; and
scoring, by the digital content summarization system server, the plurality of segments of the first media content item, for each segment of the plurality of segments, the scoring comprising determining a measure of similarity of the segment to the descriptive information using the shared dictionary.
8. The method of claim 7 , the plurality of canonical patterns appear in both the first media content item and the auxiliary data.
9. The method of claim 7 , the determining further comprising:
determining a first set of coefficients for use with the shared dictionary in approximating the plurality of feature descriptor values of each unit of the plurality of units of the first media content item;
determining a second set of coefficients for use with the shared dictionary in approximating the plurality of feature descriptor values of each second media content item of the plurality of second media content items of the auxiliary data; and
determining third and fourth sets of coefficients, the third set of coefficients for use with the plurality of feature descriptor values of each unit of the plurality of units and the fourth set of coefficients for use with the plurality of feature descriptor values of each second media content item of the plurality of second media content items of auxiliary data in approximating the shared dictionary.
10. The method of claim 9 , the scoring further comprising:
scoring each unit of the plurality of units of the first media content item, the scoring comprising determining a unit-level score for each unit representing the unit's measure of similarity to the first media content item's descriptive information using a plurality of coefficients from the first and third sets of coefficients.
11. The method of claim 7 , the determining further comprising:
learning the shared dictionary's canonical patterns such that each unit of the plurality of units of the first media content item and each second media content item of the plurality of second media content items of the auxiliary data is independently approximated by a combination of the plurality of canonical patterns of the shared dictionary and such that each canonical pattern of the plurality of canonical patterns of the shared dictionary is jointly approximated by a combination of the plurality of units of the first media content item and the plurality of second media content items of the auxiliary data.
12. The method of claim 11 , each unit of the first media content item and each second media content item of the plurality of second media content items of the auxiliary data is independently approximated by a convex combination of the plurality of canonical patterns of the shared dictionary and each canonical pattern of the plurality of canonical patterns of the shared dictionary is jointly approximated by a convex combination of the plurality of units of the first media content item and the plurality of second media content items of the auxiliary data.
13. A digital content summarization system server comprising:
a processor and a non-transitory storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic comprising:
obtaining logic executed by the processor for obtaining, for a first media content item having associated descriptive information, a plurality of second media content items as auxiliary data, the obtaining using the descriptive information associated with the first media content item, the first media content item comprising a plurality of units;
generating logic executed by the processor for generating a media content item feature space comprising a first number of feature vectors comprising feature descriptor values representing the first media content item, the generating further comprising generating an auxiliary data feature space comprising a second number of feature vectors comprising feature descriptor values representing the plurality of second media content items as the auxiliary data, obtained using the descriptive information;
identifying logic executed by the processor for identifying a plurality of segments of the first media content item, each segment comprising at least one unit of the first media content item's plurality of units;
scoring logic executed by the processor for scoring each segment of the plurality of segments of the first media content item, scoring a segment of the plurality of segments comprising determining a distance between the auxiliary data feature space and at least one feature vector of the first number of feature vectors, the at least one feature vector representing the segment, each segment's score representing a measure of similarity of the segment to the descriptive information represented by the auxiliary data feature space;
identifying logic executed by the processor for identifying at least one segment of the plurality of segments of the first media content item as more similar to the descriptive information, based on its determined distance to the auxiliary data feature space representing the descriptive information, relative to others of the plurality of segments using the scoring of the plurality of segments; and
generating logic executed by the processor for generating a summary of the first media content item, the summary comprising the at least one segment of the plurality of segments of the first media content item, identified as being more similar to the descriptive information.
14. The digital content summarization system server of claim 13 , the descriptive information comprising a title of the first media content item.
15. The digital content summarization system server of claim 13 , the scoring logic for scoring further comprising:
determining logic executed by the processor for determining, for each segment, a unit-level score for each of the at least one unit of the segment t, determining the unit-level score for a unit comprising determining a distance between the auxiliary data feature space and a feature vector of the first number of feature vectors, the feature vector representing the unit, each unit level score representing a measure of similarity of a respective one of the at least one unit to the descriptive information represented by the auxiliary data feature space; and
scoring logic executed by the processor for scoring each segment using the unit-level score determined for each of the at least one unit of the segment.
16. The digital content summarization system server of claim 15 , each segment's score comprising an average unit-level score determined using each unit-level score determined for each of the at least one unit of the segment.
17. The digital content summarization system server of claim 15 , the first media content item is a video content item, each unit of the first media content item is a frame and each segment comprises at least one frame.
18. The digital content summarization system server of claim 13 , the generating logic for generating a media content item feature space and an auxiliary data feature space further comprising:
generating logic executed by the processor for generating the media content item feature space by generating, for each of the plurality of units of the first media content item, a plurality of feature descriptor values, each feature descriptor value corresponding to a feature of a set of features; and
generating logic executed by the processor for generating the auxiliary data feature space by generating, for each second media content item of the plurality of second media content items as the auxiliary data, a plurality of feature vectors each comprising a plurality of feature descriptor values, each feature descriptor value of the plurality of feature descriptor values corresponding to a feature of the set of features used in generating the plurality of feature descriptor values for each unit of the second media content item.
19. The digital content summarization system server of claim 13 , the scoring logic further comprising:
determining logic executed by the processor for determining, using the media content item feature space and the auxiliary data feature space, a shared dictionary comprising a plurality of canonical patterns shared by the first media content item and the auxiliary data; and
scoring logic executed by the processor for scoring the plurality of segments of the first media content item, for each segment of the plurality of segments, the scoring comprising determining a measure of similarity of the segment to the descriptive information using the shared dictionary.
20. The digital content summarization system server of claim 19 , the plurality of canonical patterns appear in both the first media content item and the auxiliary data.
21. The digital content summarization system server of claim 19 , the determining logic for determining a shared dictionary further comprising:
determining logic executed by the processor for determining a first set of coefficients for use with the shared dictionary in approximating the plurality of feature descriptor values of each unit of the plurality of units of the first media content item;
determining logic executed by the processor for determining a second set of coefficients for use with the shared dictionary in approximating the plurality of feature descriptor values of each second media content item of the plurality of second media content items of the auxiliary data; and
determining logic executed by the processor for determining third and fourth sets of coefficients, the third set of coefficients for use with the plurality of feature descriptor values of each unit of the plurality of units and the fourth set of coefficients for use with the plurality of feature descriptor values of each second media content item of the plurality of second media content items of auxiliary data in approximating the shared dictionary.
22. The digital content summarization system server of claim 21 , the scoring logic for scoring the plurality of segments of the media content item further comprising:
scoring logic executed by the processor for scoring each unit of the plurality of units of the first media content item, comprising determining logic executed by the one or more processors for determining a unit-level score for each unit representing the unit's measure of similarity to the first media content item's descriptive information using a plurality of coefficients from the first and third sets of coefficients.
23. The digital content summarization system server of claim 19 , the determining logic for determining the shared dictionary further comprising:
learning logic executed by the processor for learning the shared dictionary's canonical patterns such that each unit of the plurality of units of the first media content item and each second media content item of the plurality of second media content items of the auxiliary data is independently approximated by a combination of the plurality of canonical patterns of the shared dictionary and such that each canonical pattern of the plurality of canonical patterns of the shared dictionary is jointly approximated by a combination of the plurality of units of the first media content item and the plurality of second media content items of the auxiliary data.
24. The digital content summarization system server of claim 23 , each unit of the first media content item and each second media content item of the plurality of second media content items of the auxiliary data is independently approximated by a convex combination of the plurality of canonical patterns of the shared dictionary and each canonical pattern of the plurality of canonical patterns of the shared dictionary is jointly approximated by a convex combination of the plurality of units of the first media content item and the plurality of second media content items of the auxiliary data.
25. A computer readable non-transitory storage medium for tangibly storing thereon computer readable instructions that when executed by a digital content summarization system server perform a method comprising:
obtaining, for a first media content item having associated descriptive information, a plurality of second media content items as auxiliary data, the obtaining using the descriptive information associated with the first media content item, the first media content item comprising a plurality of units;
generating a media content item feature space comprising a first number of feature vectors comprising feature descriptor values representing the first media content item, the generating further comprising generating an auxiliary data feature space comprising a second number of feature vectors comprising feature descriptor values representing the plurality of second media content items as auxiliary data, obtained using the descriptive information;
identifying a plurality of segments of the first media content item, each segment comprising at least one unit of the first media content item's plurality of units;
scoring each segment of the plurality of segments of the first media content item, scoring a segment of the plurality of segments comprising determining a distance between the auxiliary data feature and at least one feature vector of the first number of feature vectors, the at least one feature vector representing the segment, each segment's score representing a measure of similarity of the segment to the descriptive information represented by the auxiliary data feature space;
identifying at least one segment of the plurality of segments of the first media content item as more similar to the descriptive information, based on its determined distance to the auxiliary data feature space representing the descriptive information relative to others of the plurality of segments using the scoring of the plurality of segments; and
generating a summary of the first media content item, the summary comprising the at least one segment of the plurality of segments of the first media content item, identified as being more similar to the descriptive information.
26. The computer readable non-transitory storage medium of claim 25 , the descriptive information comprising a title of the first media content item.
27. The computer readable non-transitory storage medium of claim 25 , scoring further comprising:
determining, for each segment, a unit-level score for each of the at least one unit of the segment, determining the unit-level score for a unit comprising determining a distance between the auxiliary data feature space and a feature vector of the first number of feature vectors, the feature vector representing the unit, each unit level score representing a measure of similarity of a respective one of the at least one unit to the descriptive information represented by the auxiliary data feature space; and
scoring each segment using the unit-level score determined for each of the at least one unit of the segment.
28. The computer readable non-transitory storage medium of claim 27 , each segment's score comprising an average unit-level score determined using each unit-level score determined for each of the at least one unit of the segment.
29. The computer readable non-transitory storage medium of claim 27 , the first media content item is a video content item, each unit of the first media content item is a frame and each segment comprises at least one frame.
30. The computer readable non-transitory storage medium of claim 25 , generating a media content item feature space and an auxiliary data feature space further comprising:
generating the media content item feature space by generating, for each of the plurality of units of the first media content item, a plurality of feature descriptor values, each feature descriptor value corresponding to a feature of a set of features; and
generating the auxiliary data feature space by generating, for each second media content item of the plurality of second media content items as the auxiliary data, a plurality of feature vectors each comprising a plurality of feature descriptor values, each feature descriptor value of the plurality of feature descriptor values corresponding to a feature of the set of features used in generating the plurality of feature descriptor values for each unit of the second media content item.
31. The computer readable non-transitory storage medium of claim 25 , scoring further comprising:
determining, using the media content item feature space and the auxiliary data feature space, a shared dictionary comprising a plurality of canonical patterns shared by the first media content item and the auxiliary data; and
scoring the plurality of segments of the first media content item, for each segment of the plurality of segments, the scoring comprising determining a measure of similarity of the segment to the descriptive information using the shared dictionary.
32. The computer readable non-transitory storage medium of claim 31 , the plurality of canonical patterns appear in both the first media content item and the auxiliary data.
33. The computer readable non-transitory storage medium of claim 31 , determining a shared library further comprising:
determining a first set of coefficients for use with the shared dictionary in approximating the plurality of feature descriptor values of each unit of the plurality of units of the first media content item;
determining a second set of coefficients for use with the shared dictionary in approximating the plurality of feature descriptor values of each second media content item of the plurality of second media content items of the auxiliary data; and
determining third and fourth sets of coefficients, the third set of coefficients for use with the plurality of feature descriptor values of each unit of the plurality of units and the fourth set of coefficients for use with the plurality of feature descriptor values of each second media content item of the plurality of second media content items of auxiliary data in approximating the shared dictionary.
34. The computer readable non-transitory storage medium of claim 33 , scoring the plurality of segments of the media content item further comprising:
scoring each unit of the plurality of units of the first media content item, the scoring comprising determining a unit-level score for each unit representing the unit's measure of similarity to the first media content item's descriptive information using a plurality of coefficients from the first and third sets of coefficients.
35. The computer readable non-transitory storage medium of claim 31 , determining a shared library further comprising:
learning the shared dictionary's canonical patterns such that each unit of the plurality of units of the first media content item and each second media content item of the plurality of second media content items of the auxiliary data is independently approximated by a combination of the plurality of canonical patterns of the shared dictionary and such that each canonical pattern of the plurality of canonical patterns of the shared dictionary is jointly approximated by a combination of the plurality of units of the first media content item and the plurality of second media content items of the auxiliary data.
36. The computer readable non-transitory storage medium of claim 35 , each unit of the first media content item and each second media content item of the plurality of second media content items of the auxiliary data is independently approximated by a convex combination of the plurality of canonical patterns of the shared dictionary and each canonical pattern of the plurality of canonical patterns of the shared dictionary is jointly approximated by a convex combination of the plurality of units of the first media content item and the plurality of second media content items of the auxiliary data.