IP Library Granted Patent US 11,170,270
Granted Patent B2
US 11,170,270 · App. 16/656,389 · Granted Nov 9, 2021

Automatic generation of content using multimedia

Inventors: Michele Merler (Yorktown Heights, NY); Mauro Martino (Cambridge, MA); Cicero Nogueira Dos Santos (Montclair, NJ); Alfio Massimiliano Gliozzo (Brooklyn, NY); John R. Smith (New York, NY)
Assignee: International Business Machines Corporation
G06K9/6281G06N3/04G06N3/08G06N5/04G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,170,270
App. No.
16/656,389
Granted
Nov 9, 2021
Kind
B2
Abstract

Techniques for content generation are provided. A plurality of discriminative terms is determined based at least in part on a first plurality of documents that are related to a first concept, and a plurality of positive exemplars and a plurality of negative exemplars are identified using the plurality of discriminative terms. A first machine learning (ML) model is trained to classify images into concepts, based on the plurality of positive exemplars and the plurality of negative exemplars. A second concept related to the first concept is then determined, based on the first ML model. A second ML model is trained to generate images based on the second concept, and a first image is generated using the second ML model. The first image is then refined using a style transfer ML model that was trained using a plurality of style images.

Claims (89)

1. A method, comprising:

determining a plurality of discriminative terms based at least in part on a first plurality of documents that are related to a first concept;

identifying a plurality of positive exemplars and a plurality of negative exemplars using the plurality of discriminative terms;

training a first machine learning (ML) model to classify images into concepts, based on the plurality of positive exemplars and the plurality of negative exemplars;

determining a second concept related to the first concept, based on the first ML model;

training, based on the second concept, a second ML model to generate images;

generating a first image using the second ML model; and

refining the first image using a style transfer ML model that was trained using a plurality of style images.

2. The method of claim 1 , wherein determining the plurality of discriminative terms comprises:

searching one or more document repositories to retrieve the first plurality of documents that are related to the first concept;

searching the one or more document repositories to retrieve a second plurality of documents that are unrelated to the first concept; and

training a discriminative text model based on the first and second pluralities of documents.

3. The method of claim 1 , wherein identifying the plurality of positive exemplars and the plurality of negative exemplars comprises:

retrieving the plurality of positive exemplars by performing one or more searches using a first subset of the plurality of discriminative terms, wherein the first subset comprises terms that are associated with the first concept; and

retrieving the plurality of negative exemplars by performing one or more searches using a second subset of the plurality of discriminative terms, wherein the second subset comprises terms that are not associated with the first concept.

4. The method of claim 1 , wherein training the first ML model comprises:

training the first ML model to classify each of the plurality of positive exemplars as the first concept; and

training the first ML model to classify each of the plurality of negative exemplars as not the first concept, wherein the first ML model is thereby trained to generate scores for input images based on how closely they relate to the first concept.

5. The method of claim 1 , wherein determining the second concept related to the first concept comprises:

extracting a plurality of seed images from the first plurality of documents;

ranking the plurality of seed images by evaluating each respective seed image of the plurality of seed images using the first ML model;

selecting one or more of the plurality of seed images based on the ranking; and

identifying one or more concepts reflected in the selected one or more of the plurality of seed images.

6. The method of claim 1 , wherein training the second ML model comprises:

retrieving a plurality of images by performing one or more searches using the second concept; and

training a generative adversarial network (GAN) using the plurality of images.

7. The method of claim 1 , wherein the style transfer ML model was trained by:

receiving the plurality of style images, wherein the plurality of style images reflect a desired visual style of the first image;

tiling the plurality of style images to create a single style reference image; and

training the style transfer ML model using the single style reference image.

8. A computer-readable storage medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising:

determining a plurality of discriminative terms based at least in part on a first plurality of documents that are related to a first concept;

identifying a plurality of positive exemplars and a plurality of negative exemplars using the plurality of discriminative terms;

training a first machine learning (ML) model to classify images into concepts, based on the plurality of positive exemplars and the plurality of negative exemplars;

determining a second concept related to the first concept, based on the first ML model;

training, based on the second concept, a second ML model to generate images;

generating a first image using the second ML model; and

refining the first image using a style transfer ML model that was trained using a plurality of style images.

9. The computer-readable storage medium of claim 8 , wherein determining the plurality of discriminative terms comprises:

searching one or more document repositories to retrieve the first plurality of documents that are related to the first concept;

searching the one or more document repositories to retrieve a second plurality of documents that are unrelated to the first concept; and

training a discriminative text model based on the first and second pluralities of documents.

10. The computer-readable storage medium of claim 8 , wherein identifying the plurality of positive exemplars and the plurality of negative exemplars comprises:

retrieving the plurality of positive exemplars by performing one or more searches using a first subset of the plurality of discriminative terms, wherein the first subset comprises terms that are associated with the first concept; and

retrieving the plurality of negative exemplars by performing one or more searches using a second subset of the plurality of discriminative terms, wherein the second subset comprises terms that are not associated with the first concept.

11. The computer-readable storage medium of claim 8 , wherein training the first ML model comprises:

training the first ML model to classify each of the plurality of positive exemplars as the first concept; and

training the first ML model to classify each of the plurality of negative exemplars as not the first concept, wherein the first ML model is thereby trained to generate scores for input images based on how closely they relate to the first concept.

12. The computer-readable storage medium of claim 8 , wherein determining the second concept related to the first concept comprises:

extracting a plurality of seed images from the first plurality of documents;

ranking the plurality of seed images by evaluating each respective seed image of the plurality of seed images using the first ML model;

selecting one or more of the plurality of seed images based on the ranking; and

identifying one or more concepts reflected in the selected one or more of the plurality of seed images.

13. The computer-readable storage medium of claim 8 , wherein training the second ML model comprises:

retrieving a plurality of images by performing one or more searches using the second concept; and

training a generative adversarial network (GAN) using the plurality of images.

14. The computer-readable storage medium of claim 8 , wherein the style transfer ML model was trained by:

receiving the plurality of style images, wherein the plurality of style images reflect a desired visual style of the first image;

tiling the plurality of style images to create a single style reference image; and

training the style transfer ML model using the single style reference image.

15. A system comprising:

one or more computer processors; and

a memory containing a program which when executed by the one or more computer processors performs an operation, the operation comprising:

determining a plurality of discriminative terms based at least in part on a first plurality of documents that are related to a first concept;

identifying a plurality of positive exemplars and a plurality of negative exemplars using the plurality of discriminative terms;

training a first machine learning (ML) model to classify images into concepts, based on the plurality of positive exemplars and the plurality of negative exemplars;

determining a second concept related to the first concept, based on the first ML model;

training, based on the second concept, a second ML model to generate images;

generating a first image using the second ML model; and

refining the first image using a style transfer ML model that was trained using a plurality of style images.

16. The system of claim 15 , wherein determining the plurality of discriminative terms comprises:

searching one or more document repositories to retrieve the first plurality of documents that are related to the first concept;

searching the one or more document repositories to retrieve a second plurality of documents that are unrelated to the first concept; and

training a discriminative text model based on the first and second pluralities of documents.

17. The system of claim 15 , wherein identifying the plurality of positive exemplars and the plurality of negative exemplars comprises:

retrieving the plurality of positive exemplars by performing one or more searches using a first subset of the plurality of discriminative terms, wherein the first subset comprises terms that are associated with the first concept; and

retrieving the plurality of negative exemplars by performing one or more searches using a second subset of the plurality of discriminative terms, wherein the second subset comprises terms that are not associated with the first concept.

18. The system of claim 15 , wherein training the first ML model comprises:

training the first ML model to classify each of the plurality of positive exemplars as the first concept; and

training the first ML model to classify each of the plurality of negative exemplars as not the first concept, wherein the first ML model is thereby trained to generate scores for input images based on how closely they relate to the first concept.

19. The system of claim 15 , wherein determining the second concept related to the first concept comprises:

extracting a plurality of seed images from the first plurality of documents;

ranking the plurality of seed images by evaluating each respective seed image of the plurality of seed images using the first ML model;

selecting one or more of the plurality of seed images based on the ranking; and

identifying one or more concepts reflected in the selected one or more of the plurality of seed images.

20. The system of claim 15 , wherein the style transfer ML model was trained by:

receiving the plurality of style images, wherein the plurality of style images reflect a desired visual style of the first image;

tiling the plurality of style images to create a single style reference image; and

training the style transfer ML model using the single style reference image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 29, 2019
From: MERLER, MICHELE; MARTINO, MAURO; NOGUEIRA DOS SANTOS, CICERO; GLIOZZO, ALFIO MASSIMILIANO; SMITH, JOHN R
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 050850/0348 →
Continuity (1)
Related Publication 20210117736A1 · Apr 22, 2021
Cited By (1)
US 12,657,879