IP Library Granted Patent US 10,503,738
Granted Patent B2
US 10,503,738 · App. 15/162,947 · Granted Dec 10, 2019

Generating recommendations for media assets to be displayed with related text content

Inventors: Harsh Jhamtani (Kanpur, IN); Siddhartha Kumar Dutta (Thane, IN); Midhun Gundapuneni (Hyderabad, IN); Shubham Varma (Ahmedabad, IN); Cedric Huesler (San Jose, CA)
Assignee: Adobe Inc.
G06F16/24578G06F16/40G06F17/241G06F17/2785
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,503,738
App. No.
15/162,947
Granted
Dec 10, 2019
Kind
B2
Abstract

Certain embodiments involve creating multimedia content with text and media assets that illustrate the text. Multiple sentences are ranked on various features. A sentence ranking is determined based on, for example, the presence of important phrases in the sentence, the degree to which informational content of the sentence can be represented through a media asset, the presence of one or more sentiments associated with the sentence, and the readability of the sentence. In some examples, the ranked sentences are analyzed to determine similar information content, and the sentences are re-ranked based on this analysis. A subset of the ranked sentences with higher ranks are analyzed to determine similarities between content in the sentence and text descriptions of media assets. This analysis can be used to select appropriate images or other media assets. Multimedia content is generated in which the selected media assets are positioned near the set of sentences.

Claims (101)

1. A method of creating multimedia content comprising both media assets and text describing the media assets, the method comprising:

receiving, by a processing device, a set of sentences and a threshold number of sentences in the set to be accompanied by a media asset;

determining, by the processing device and for each sentence in the set, a respective rank based on a combination of features;

ranking the sentences within the set based on the determined ranks for the sentences;

selecting, by the processing device, a first ranked sentence having a ranking below the threshold number, a second ranked sentence having a ranking above the threshold number, and a third ranked sentence having a ranking above the threshold number;

determining, by the processing device, a first degree of overlap between the first sentence and the second sentence, wherein the first degree of overlap is determined based on a first cosine similarity between a first vector space representation associated with the first sentence and a second vector space representation associated with the second sentence;

determining, by the processing device, a second degree of overlap between the first sentence and the third sentence, wherein the second degree of overlap is determined based on a second cosine similarity between the first vector space representation and a third vector space representation associated with the third sentence;

responsive to determining that the first degree of overlap is less than the second degree of overlap, performing a subsequent ranking operation on the ranked sentences, wherein the first ranked sentence is subsequently ranked above the second ranked sentence;

selecting, by the processing device, a sub-set of the sentences based upon the subsequent ranking operation; and

generating, by the processing device, content comprising each respective sentence in the sub-set of sentences co-located with a respective media asset associated with the respective sentence.

2. The method of claim 1 , wherein the combination of features comprises at least two of:

a phrase having a threshold importance to semantic content of the sentence,

a picturability of the sentence indicating a degree to which information in the sentence can be represented by a particular media asset,

a sentiment score for the sentence, or

a readability rating for the sentence.

3. The method of claim 2 , wherein determining the respective rank further comprises:

receiving an additional set of sentences each having a respective previously determined rank based on a group of features comprising at least two of additional identified phrases having the threshold importance, an additional picturability, an additional sentiment score, and an readability rating; and

evaluating the additional set of sentences to determine a relative importance of each feature in the group of features, wherein the respective rank for each sentence in the set of sentences is determined based on the determined relative importance.

4. The method of claim 1 , wherein:

determining the respective rank further comprises identifying a degree to which each sentence in the set of sentences requires an accompanying media asset based on the combination of features; and

ranking the sentences further comprises classifying, based on the identified degree and the received threshold, whether each sentence in the set of sentences requires an accompanying media asset.

5. The method of claim 4 , wherein classifying whether each sentence in the set of sentences requires an accompanying media asset further comprises:

for each sentence, providing additional values corresponding to each feature of the combination of features;

based on the additional values for sentences in the set of sentences, determining an error for each feature of the combination of features;

comparing the determined errors; and

determining that a first feature having larger error relative to a second feature has a higher relative importance to classifying sentences.

6. The method of claim 1 , wherein determining the respective rank further comprises:

for each sentence in the set of sentences, providing additional values corresponding to each feature of the combination of features;

based on the additional values for sentences in the set of sentences, determining coefficients for a regression model;

for each sentence in the set of sentences, determining an aggregate rating based upon the regression mode; and

prioritizing a first sentence with a relatively higher aggregate rating over a second sentence with a relatively lower aggregate rating.

7. The method of claim 1 , further comprising displaying the generated content having the respective media asset next to the respective sentence.

8. The method of claim 1 , wherein generating the content comprises:

accessing a repository containing media assets, each media asset having associated text;

comparing the respective sentence to the associated text of at least two of the media assets in the repository; and

selecting a given media asset based on a similarity between the respective sentence to the associated text of the given media asset.

9. A system comprising:

a repository containing media assets,

a memory, and

a processor for executing instructions stored in a computer-readable medium, wherein when executed by the processor, the instructions perform operations comprising:

receiving a set of sentences and a threshold number of sentences in the set to be accompanied by a media asset;

determining, for each sentence in the set of sentences, a respective rank, wherein determining the respective rank is determined based on at least two of:

a phrase having a threshold importance to semantic content of the sentence,

a picturability of the sentence indicating a degree to which information in the sentence can be represented by a particular media asset,

a sentiment score for the sentence, or

a readability rating for the sentence, and

ranking the sentences within the set based on the determined ranks for the sentences;

selecting a first sub-set of the sentences having a rank above the received threshold number of sentences;

selecting a first ranked sentence in the set of sentences, the first sentence having a ranking below the threshold number;

selecting a second ranked sentence in the first sub-set of the sentences, the second sentence having a ranking above the threshold number;

selecting a third ranked sentence in the first sub-set of the sentences, the third sentence having a ranking above the threshold number;

determining a first degree of overlap between the first sentence and the second sentence, wherein the first degree of overlap is determined based on a first cosine similarity between a first vector space representation associated with the first sentence and a second vector space representation associated with the second sentence;

determining a second degree of overlap between the first sentence and the third sentence, wherein the second degree of overlap is determined based on a second cosine similarity between the first vector space representation and a second vector space representation associated with the third sentence; and

responsive to determining that the first degree of overlap is less than the second degree of overlap, performing a subsequent ranking operation on the ranked sentences, wherein the first ranked sentence is subsequently ranked above the second ranked sentence;

selecting a second sub-set of the sentences based upon the subsequent ranking operation; and

selecting, from the repository and based on the second sub-set of the sentences, one or more of the media assets for inclusion in a web page or other electronic content.

10. The system of claim 9 , wherein determining the respective rank further comprises:

receiving an additional set of sentences each having a respective previously determined rank based on a group of features comprising at least two of:

additional identified phrases having threshold degrees of importance in the additional set of sentences,

an additional picturability for one or more of the additional set of sentences,

an additional sentiment score for one or more of additional set of sentences, and

an readability rating for one or more of additional set of sentences;

evaluating the additional set of sentences to determine a relative importance of each feature in the group of features; and

determining the respective rank based in part upon the determined relative importance of each feature in the group of features.

11. The system of claim 9 , wherein determining the respective rank further comprises identifying a degree to which each sentence in the set requires an accompanying media asset based on the picturability, the phrase, the sentiment score, or the readability rating, and

wherein ranking the sentences further comprises classifying, based on the identified degree and the received threshold, whether each sentence in the set requires an accompanying media asset.

12. The system of claim 11 , wherein classifying whether each sentence in the set requires an accompanying media asset further comprises:

for each sentence, providing additional values corresponding to each of the picturability, the phrase, the sentiment score, or the readability rating;

based on the additional values for sentences in the set, determining an error for each of the picturability, the phrase, the sentiment score, or the readability rating;

comparing the determined errors; and

determining that a first error having larger value relative to a second error has a higher relative importance to classifying sentences.

13. The system of claim 9 , wherein determining the respective rank further comprises:

for each sentence in the set of sentences, providing additional values corresponding to each of the picturability, the phrase, the sentiment score, or the readability rating;

based on the additional values for sentences in the set of sentences, determining coefficients for a regression model;

for each sentence in the set of sentences, determining an aggregate rating based upon the regression model; and

determining the respective rank comprises prioritizing a first sentence with a relatively higher aggregate rating over a second sentence with a relatively lower aggregate rating.

14. The system of claim 9 , the operations further comprising generating the web page or other electronic content comprising each of the second sub-set of the sentences co-located with a respective media asset associated with one or more sentences of the second sub-set of the sentences.

15. A non-transitory computer-readable medium on which is encoded program code for creating multimedia content comprising both media assets and text describing the media assets, the program code comprising:

program code for receiving, by a processing device, a set of sentences and a threshold number of sentences in the set to be accompanied by a media asset;

program code for determining, by the processing device and for each sentence in the set, a respective rank based on a combination of features;

program code for ranking the sentences within the set based on the determined ranks for the sentences;

program code for selecting a first ranked sentence having a ranking below the threshold number, a second ranked sentence having a ranking above the threshold number, and a third ranked sentence having a ranking above the threshold number;

program code for determining a first degree of overlap between the first sentence and the second sentence, wherein the first degree of overlap is determined based on a first cosine similarity between a first vector space representation associated with the first sentence and a second vector space representation associated with the second sentence;

program code for determining a second degree of overlap between the first sentence and the third sentence, wherein the second degree of overlap is determined based on a second cosine similarity between the first vector space representation and a third vector space representation associated with the third sentence; and

program code for, responsive to determining that the first degree of overlap is less than the second degree of overlap, performing a subsequent ranking operation on the ranked sentences, wherein the first ranked sentence is subsequently ranked above the second ranked sentence;

program code for selecting, by the processing device, a sub-set of the sentences based upon the subsequent ranking operation; and

program code for generating, by the processing device, content comprising each respective sentence in the sub-set of sentences co-located with a respective media asset associated with the respective sentence.

16. The non-transitory computer-readable medium of claim 15 , wherein the combination of features comprises at least two of:

a phrase having a threshold importance to semantic content of the sentence,

a picturability of the sentence indicating a degree to which information in the sentence can be represented by a particular media asset,

a sentiment score for the sentence, or

a readability rating for the sentence.

17. The non-transitory computer-readable medium of claim 16 , wherein determining the respective rank further comprises:

program code for receiving an additional set of sentences each having a respective previously determined rank based on a group of features comprising at least two of additional identified phrases having the threshold importance, an additional picturability, an additional sentiment score, and a readability rating;

program code for evaluating the additional set of sentences to determine a relative importance of each feature in the group of features; and

program code for determining the respective rank based in part upon the determined relative importance of each feature in the group of features.

18. The non-transitory computer-readable medium of claim 15 , wherein determining the respective rank further comprises:

program code for providing for each sentence in the set of sentences, additional values corresponding to each feature of the combination of features;

program code for determining, based on the additional values for sentences in the set of sentences, coefficients for a regression model;

program code for determining, for each sentence in the set of sentences, an aggregate rating based upon the regression model; and

program code for prioritizing a first sentence with a relatively higher aggregate rating over a second sentence with a relatively lower aggregate rating.

Assignments (2)
CHANGE OF NAME Recorded Mar 6, 2019
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 048525/0042 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 24, 2016
From: JHAMTANI, HARSH; HUESLER, CEDRIC; VARMA, SHUBHAM; DUTTA, SIDDHARTHA KUMAR; GUNDAPUNENI, MIDHUN
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 038701/0340 →
Continuity (2)
Provisional Application 62310040 · Mar 18, 2016
Related Publication 20170270123A1 · Sep 21, 2017
Cited By (2)
US 12,242,515 US 12,657,940