IP Library Granted Patent US 11,164,021
Granted Patent B2
US 11,164,021 · App. 16/875,680 · Granted Nov 2, 2021

Methods, systems, and media for discriminating and generating translated images

Inventors: Md Mahfuzur Rahman Siddiquee (Tempe, AZ); Zongwei Zhou (Tempe, AZ); Ruibin Feng (Tempe, AZ); Nima Tajbakhsh (Los Angeles, CA); Jianming Liang (Scottsdale, AZ)
Assignee: Arizona Board of Regents on behalf of Arizona State University
G06K9/00899G06K9/00362G06K9/6256
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,164,021
App. No.
16/875,680
Granted
Nov 2, 2021
Kind
B2
Abstract

Methods, systems, and media for discriminating and generating translated images are provided. In some embodiments, the method comprises: identifying a set of training images, wherein each image is associated with at least one domain from a plurality of domains; training a generator network to generate: i) a first fake image that is associated with a first domain; and ii) a second fake image that is associated with a second domain; training a discriminator network, using as inputs to the discriminator network: i) an image from the set of training images; ii) the first fake image; and iii) the second fake image; and using the generator network to generate, for an image not included in the set of training images at least one of: i) a third fake image that is associated with the first domain; and ii) a fourth fake image that is associated with the second domain.

Claims (32)

1. A method for discriminating and generating translated images, comprising:

identifying a set of training images, wherein each image in the set of training images is associated with at least one domain from a plurality of domains, and wherein each domain corresponds to an attribute of an image;

training a generator network of a generative adversarial network to generate, using a first image from the set of training images that is associated with a first domain from the plurality of domains: i) a first fake image that is associated with the first domain from the plurality of domains; and ii) a second fake image that is associated with a second domain from the plurality of domains;

training a discriminator network of the generative adversarial network using as inputs to the discriminator network: i) an image from the set of training images; ii) the first fake image that is associated with the first domain from the plurality of domains; and iii) the second fake image that is associated with the second domain from the plurality of domains; and

in response to determining that training of the generator network and the discriminator network is complete, using the generator network to generate, for an image not included in the set of training images that is associated with the first domain from the plurality of domains at least one of: i) a third fake image that is associated with the first domain from the plurality of domains; and ii) a fourth fake image that is associated with the second domain from the plurality of domains.

2. The method of claim 1 , wherein each image in the set of training images corresponds to an image of a portion of a body of a patient, and wherein each domain in the plurality of domains corresponds to a presence or an absence of a disease state in an image.

3. The method of claim 2 , wherein the first domain corresponds to a presence of a lesion in the image of the portion of the body of the patient, and wherein the second domain corresponds to a lack of a lesion in the image of the portion of the body of the patient.

4. The method of claim 3 , further comprising generating a difference map by subtracting the fourth fake image that is associated with the second domain from the image not included in the training set.

5. The method of claim 1 , wherein training the generator network comprises optimizing a generator loss function, and wherein the generator loss function includes a penalty based on the generated first fake image associated with the first domain of the plurality of domains.

6. The method of claim 1 , wherein training the generator network comprises optimizing a generator loss function, and wherein the generator loss function includes a penalty based on the generated second fake image associated with the second domain of the plurality of domains.

7. A system for discriminating and generating translated images, the system comprising:

a memory; and

a hardware processor that, when executing computer-executable instructions stored in the memory, is configured to:

identify a set of training images, wherein each image in the set of training images is associated with at least one domain from a plurality of domains, and wherein each domain corresponds to an attribute of an image;

train a generator network of a generative adversarial network to generate, using a first image from the set of training images that is associated with a first domain from the plurality of domains: i) a first fake image that is associated with the first domain from the plurality of domains; and ii) a second fake image that is associated with a second domain from the plurality of domains;

train a discriminator network of the generative adversarial network using as inputs to the discriminator network: i) an image from the set of training images; ii) the first fake image that is associated with the first domain from the plurality of domains; and iii) the second fake image that is associated with the second domain from the plurality of domains; and

in response to determining that training of the generator network and the discriminator network is complete, use the generator network to generate, for an image not included in the set of training images that is associated with the first domain from the plurality of domains at least one of: i) a third fake image that is associated with the first domain from the plurality of domains; and ii) a fourth fake image that is associated with the second domain from the plurality of domains.

8. The system of claim 7 , wherein each image in the set of training images corresponds to an image of a portion of a body of a patient, and wherein each domain in the plurality of domains corresponds to a presence or an absence of a disease state in an image.

9. The system of claim 8 , wherein the first domain corresponds to a presence of a lesion in the image of the portion of the body of the patient, and wherein the second domain corresponds to a lack of a lesion in the image of the portion of the body of the patient.

10. The system of claim 9 , wherein the hardware processor is further configured to generate a difference map by subtracting the fourth fake image that is associated with the second domain from the image not included in the training set.

11. The system of claim 7 , wherein training the generator network comprises optimizing a generator loss function, and wherein the generator loss function includes a penalty based on the generated first fake image associated with the first domain of the plurality of domains.

12. The system of claim 7 , wherein training the generator network comprises optimizing a generator loss function, and wherein the generator loss function includes a penalty based on the generated second fake image associated with the second domain of the plurality of domains.

13. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for discriminating and generating translated images, the method comprising:

identifying a set of training images, wherein each image in the set of training images is associated with at least one domain from a plurality of domains, and wherein each domain corresponds to an attribute of an image;

training a generator network of a generative adversarial network to generate, using a first image from the set of training images that is associated with a first domain from the plurality of domains: i) a first fake image that is associated with the first domain from the plurality of domains; and ii) a second fake image that is associated with a second domain from the plurality of domains;

training a discriminator network of the generative adversarial network using as inputs to the discriminator network: i) an image from the set of training images; ii) the first fake image that is associated with the first domain from the plurality of domains; and iii) the second fake image that is associated with the second domain from the plurality of domains; and

in response to determining that training of the generator network and the discriminator network is complete, using the generator network to generate, for an image not included in the set of training images that is associated with the first domain from the plurality of domains at least one of: i) a third fake image that is associated with the first domain from the plurality of domains; and ii) a fourth fake image that is associated with the second domain from the plurality of domains.

14. The non-transitory computer-readable medium of claim 13 , wherein each image in the set of training images corresponds to an image of a portion of a body of a patient, and wherein each domain in the plurality of domains corresponds to a presence or an absence of a disease state in an image.

15. The non-transitory computer-readable medium of claim 14 , wherein the first domain corresponds to a presence of a lesion in the image of the portion of the body of the patient, and wherein the second domain corresponds to a lack of a lesion in the image of the portion of the body of the patient.

16. The non-transitory computer-readable medium of claim 15 , wherein the method further comprises generating a difference map by subtracting the fourth fake image that is associated with the second domain from the image not included in the training set.

17. The non-transitory computer-readable medium of claim 13 , wherein training the generator network comprises optimizing a generator loss function, and wherein the generator loss function includes a penalty based on the generated first fake image associated with the first domain of the plurality of domains.

18. The non-transitory computer-readable medium of claim 13 , wherein training the generator network comprises optimizing a generator loss function, and wherein the generator loss function includes a penalty based on the generated second fake image associated with the second domain of the plurality of domains.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 3, 2020
From: RAHMAN SIDDIQUEE, MD MAHFUZUR; ZHOU, ZONGWEI; FENG, RUIBIN; TAJBAKHSH, NIMA; LIANG, JIANMING
To: ARIZONA BOARD OF REGENTS ON BEHALF OF ARIZONA STATE UNIVERSITY
Reel/Frame 054533/0938 →
Continuity (2)
Provisional Application 62849030 · May 16, 2019
Related Publication 20200364477A1 · Nov 19, 2020