IP Library › Granted Patent US 12,321,420
Granted Patent B2
US 12,321,420 · App. 18/652,226 · Granted Jun 3, 2025

Systems and methods for training generative adversarial networks and use of trained generative adversarial networks

Inventors: Nhan Ngo Dinh (Rome, IT); Giulio Evangelisti (Rome, IT); Flavio Navari (Rome, IT)
Assignee: COSMO ARTIFICIAL INTELLIGENCE—AI LIMITED
G06F18/2413A61B1/000096A61B1/273A61B1/2736A61B1/31G06F18/214G06F18/2148G06F18/217G06F18/41G06N3/045G06N3/08G06N3/088G06T7/0012G16H30/40G16H50/20G06T2207/10016G06T2207/10068G06T2207/20081G06T2207/20084G06T2207/30032G06T2207/30096G06V2201/032
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,321,420
App. No.
18/652,226
Granted
Jun 3, 2025
Kind
B2
Abstract

The present disclosure relates to computer-implemented systems and methods for training and using generative adversarial networks. In one implementation, a system for training a generative adversarial network may include at least one processor that may provide a first plurality of images including representations of a feature-of-interest and indicators of locations of the feature-of-interest and use the first plurality and indicators to train an object detection network. Further, the processor(s) may provide a second plurality of images including representation of the feature-of-interest, and apply the trained object detection network to the second plurality to produce a plurality of detections of the feature-of-interest. Additionally, the processor(s) may provide manually set verifications of true positives and false positives with respect to the plurality of detections, use the verifications tr train a generative adversarial network, and retrain the generative adversarial network using at least one further set of images, further detections, and further manually set verifications.

Claims (37)

1. A method for training a generative adversarial network, comprising:

receiving a first plurality of feature indicators for at least one region in a first plurality of image frames, the at least one region including one or more representations of a feature-of-interest;

training a discriminator network using a first training set including the first plurality of image frames and the first plurality of feature indicators;

applying the trained discriminator network to a second plurality of image frames to produce a second plurality of feature indicators for at least one region in the second plurality of image frames, the at least one region including one or more representations of the feature-of-interest;

receiving verifications of true positives and false positives with respect to the second plurality of feature indicators; and

training a generative adversarial network using a second training set including the second plurality of image frames and the verifications.

2. The method of claim 1 , further comprising extracting the first plurality of image frames from a database.

3. The method of claim 1 , further comprising displaying, via a display device, the first plurality of image frames.

4. The method of claim 1 , wherein the first plurality of feature indicators are received from an input device.

5. The method of claim 1 , further comprising extracting the second plurality of image frames from a database.

6. The method of claim 1 , wherein the generative adversarial network includes a generative network and an adversarial network.

7. The method of claim 6 , wherein training the generative adversarial network includes training the generative network to produce artificial representations of the feature-of-interest.

8. The method of claim 7 , wherein training the generative adversarial network further includes training the adversarial network to distinguish the artificial representations of the feature-of-interest from real representations of the feature-of-interest.

9. The method of claim 6 , wherein training the generative adversarial network further comprises training the generative network to produce artificial representations of a false feature-of-interest that looks similar to a true feature-of-interest.

10. The method of claim 1 , wherein the first plurality of image frames and the second plurality of image frames comprise medical images captured by an imaging device.

11. The method of claim 1 , wherein the feature-of-interest includes an abnormality comprising at least one of a formation on or of human tissues, a change in human tissue from one type of cell to another type of cell, an absence of human tissue from a location where the human tissue is expected, and a lesion.

12. The method of claim 1 , wherein the discriminator network comprises an object detection network.

13. The method of claim 1 , wherein training the discriminator network includes adjusting one or more weights associated with one or more nodes of the discriminator network or adjusting a function associated with one or more nodes of the discriminator network.

14. The method of claim 1 , further comprising retraining the trained discriminator network using the verifications of the true positives and the false positives.

15. The method of claim 1 , wherein the verifications of the true positives and the false positives are received via an input device.

16. A non-transitory computer-readable medium comprising instructions configured to cause at least one processor to:

receive a first plurality of feature indicators for at least one region in a first plurality of image frames, the at least one region including one or more representations of a feature-of-interest;

train a discriminator network using a first training set including the first plurality of image frames and the first plurality of feature indicators;

apply the trained discriminator network to a second plurality of image frames to produce a second plurality of feature indicators for at least one region in the second plurality of image frames, the at least one region including one or more representations of the feature-of-interest;

receive verifications of true positives and false positives with respect to the second plurality of feature indicators; and

train a generative adversarial network using a second training set including the second plurality of image frames and the verifications.

17. The non-transitory computer-readable medium of claim 16 , wherein training the generative adversarial network includes training a generative network to produce artificial representations of the feature-of-interest.

18. The non-transitory computer-readable medium of claim 17 , wherein training the generative adversarial network further includes training an adversarial network to distinguish the artificial representations of the feature-of-interest from real representations of the feature-of-interest.

19. A system for training a generative adversarial network, comprising:

at least one memory configured to store instructions; and

at least one processor configured to execute the instructions to:

receive a first plurality of feature indicators for at least one region in a first plurality of image frames, the at least one region including one or more representations of a feature-of-interest;

train a discriminator network using a first training set including the first plurality of image frames and the first plurality of feature indicators;

apply the trained discriminator network to a second plurality of image frames to produce a second plurality of feature indicators for at least one region in the second plurality of image frames, the at least one region including one or more representations of the feature-of-interest;

receive verifications of true positives and false positives with respect to the second plurality of feature indicators; and

train a generative adversarial network using a second training set including the second plurality of image frames and the verifications.

20. The system of claim 19 , wherein training the generative adversarial network includes training a generative network to produce artificial representations of the feature-of-interest.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 1, 2024
From: NGO DINH, NHAN; EVANGELISTI, GIULIO; NAVARI, FLAVIO
To: LINKVERSE S.R.L.
Reel/Frame 067283/0661 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 1, 2024
From: LINKVERSE S.R.L.
To: COSMO TECHNOLOGIES LIMITED
Reel/Frame 067283/0755 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 1, 2024
From: GRANELL STRATEGIC INVESTMENT FUND LIMITED
To: COSMO ARTIFICIAL INTELLIGENCE - AI LIMITED
Reel/Frame 067284/0077 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 1, 2024
From: COSMO TECHNOLOGIES LIMITED
To: COSMO ARTIFICIAL INTELLIGENCE - AI LIMITED
Reel/Frame 067284/0165 →
Priority Claims (1)
EP 18180570 · Jun 28, 2018 · regional
Continuity (3)
Continuation 17251773
Continuation 16008006 · Jun 13, 2018
Related Publication 20240303299A1 · Sep 12, 2024
References Cited (85)
US 9530205B2 · Kim et al. · 2016 [cited by applicant]
US 10346740B2 · Zhang et al. · 2019 [cited by applicant]
US 10395356B2 · Zhang et al. · 2019 [cited by applicant]
US 10540578B2 · Madani · 2020 [cited by examiner]
US 10799098B2 · Oosake · 2020 [cited by applicant]
US 10810460B2 · Ngo Dinh et al. · 2020 [cited by applicant]
US 10841514B2 · Sachdev et al. · 2020 [cited by applicant]
US 11195052B2 · Dinh et al. · 2021 [cited by applicant]
US 20080058593A1 · Gu et al. · 2008 [cited by applicant]
US 20110301447A1 · Park et al. · 2011 [cited by applicant]
US 20150374210A1 · Durr et al. · 2015 [cited by applicant]
US 20160284103A1 · Huang · 2016 [cited by applicant]
US 20170132468A1 · Mosher et al. · 2017 [cited by applicant]
US 20170265747A1 · Tajbakhsh et al. · 2017 [cited by applicant]
US 20170286805A1 · Yu et al. · 2017 [cited by applicant]
US 20170300814A1 · Shaked et al. · 2017 [cited by applicant]
US 20170316286A1 · Szegedy et al. · 2017 [cited by applicant]
US 20180012463A1 · Chaudhry et al. · 2018 [cited by applicant]
US 20180075581A1 · Shi et al. · 2018 [cited by applicant]
US 20180075599A1 · Tajbakhsh et al. · 2018 [cited by applicant]
US 20180144214A1 · Hsieh et al. · 2018 [cited by applicant]
US 20180365826A1 · Oh et al. · 2018 [cited by applicant]
US 20190164287A1 · Gregson et al. · 2019 [cited by applicant]
US 20190252073A1 · Hsu et al. · 2019 [cited by applicant]
US 20190297276A1 · Sachdev et al. · 2019 [cited by applicant]
US 20190304092A1 · Akselrod-Ballin · 2019 [cited by examiner]
US 20190311476A1 · Hayami et al. · 2019 [cited by applicant]
US 20190380617A1 · Oosake et al. · 2019 [cited by applicant]
US 20190385018A1 · Ngo Dinh et al. · 2019 [cited by applicant]
US 20190385302A1 · Ngo Dinh et al. · 2019 [cited by applicant]
US 20200008653A1 · Kamon · 2020 [cited by applicant]
US 20200069160A1 · Oosake · 2020 [cited by applicant]
US 20200226423A1 · Ngo Dinh et al. · 2020 [cited by applicant]
US 20200337537A1 · Hirasawa et al. · 2020 [cited by applicant]
US 20210022586A1 · Mori et al. · 2021 [cited by applicant]
US 20210133972A1 · Ngo Dinh et al. · 2021 [cited by applicant]
US 20210153730A1 · Karino · 2021 [cited by applicant]
US 20210153808A1 · Tada et al. · 2021 [cited by applicant]
CN 107256552A · 2017 [cited by applicant]
CN 107451619A · 2017 [cited by applicant]
CN 107451994A · 2017 [cited by applicant]
TW 201800983A · 2018 [cited by applicant]
TW 201802726A · 2018 [cited by applicant]
WO 2016161115A1 · 2016 [cited by applicant]
WO 2017042812A2 · 2017 [cited by applicant]
Ahn et al., Finding Small-Bowel Lesions: Challenges in Endoscopy-Image-Based Learning Systems, Computer, IEEE Computer Society, vol. 51, No. 5, May 24, 2018, pp. 68-76. [cited by applicant]
An Office Action and Search Report issued by the Taiwan Patent Office for Taiwan Patent Application No. 108120187, mailed on Aug. 16, 2022. [cited by applicant]
Billah et al., An Automatic Gastrointestinal Polyp Detection System in Video Endoscopy Using Fusion of Color Wavelet and Convolutional Neural Network Features, International Journal of Biomedicallmaging, vol. 2017, Aug.… [cited by applicant]
Billah et al., Gastrointestinal Polyp Detection in Endoscopic Images Using an Improved Feature Extraction Method, Biomedical Engineering Letters, vol. 8, No. 1, Sep. 7, 2017, pp. 69-75. [cited by applicant]
Brandao et al., Towards a Computed-Aided Diagnosis System in Colonoscopy: Automatic Polyp Segmentation Using Convolution Neural Networks, Journal of Medical Robotics Research, vol. 03, No. 02, Feb. 23, 2018, pp. 1840002… [cited by applicant]
Byrne et al., Artificial Intelligence (AI) in Endoscopy-Deep Learning for Optical Biopsy of Colorectal Polyps in Real-Time on Unaltered Endoscopic Videos, Gastrointestinal Endoscopy Supplement, vol. 85, No. 5, May 2017,… [cited by applicant]
Byrne et al., Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model, Gut, Oct. 2017, 7 pages. [cited by applicant]
Byrne et al., Will Computer-Aided Detection and Diagnosis Revolutionize Colonoscopy?, Gastroenterology, vol. 152, No. 6, Oct. 2017, pp. 1460-1464 and 1464.e1. [cited by applicant]
Chen et al., Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis, Gastroenterology, vol. 154, No. 3, Feb. 2018, pp. 568-575. [cited by applicant]
Feichtenhofer et al., Detect to Track and Track to Detect, arXiv 2018, available at https://arxiv.org/abs/1710.03958, Oct. 2017, 9 pages. [cited by applicant]
Henriksen et al., Polyp Detection using Neural Networks, Master's Thesis, University of Oslo, Oslo, Norway, May 2017, 115 pages. [cited by applicant]
International Search Report, International Application No. PCT/EP2019/065256 for Cosmo Artificial Intelligence—AI Limited, mailed Sep. 17, 2019, 4 pages. [cited by applicant]
Kang et al., Object Detection from Video Tubelets with Convolutional Neural Networks, 2016 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp. 817-825. [cited by applicant]
Karnes et al., Automated Polyp Detection Using Deep Learning: Leveling the Field, Gastrointestinal Endoscopy Supplement, vol. 85, No. 5, May 2017, pp. AB376-AB377. [cited by applicant]
Li et al., Perceptual Generative Adversarial Networks for Small Object Detection, arXiv 2017, available at https ://arxiv.org/abs/1 706. 05274, Jun. 2017, 9 pages. [cited by applicant]
Lucas Hadjilucas, Framework for the Detection and Classification of Colorectal Polyps, PhD dissertation, Imperial College of Science, Technology and Medicine, London, United Kington, 2014, 142 pages. [cited by applicant]
Mahmood et al., Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy, Medical Image Analysis, vol. 48, Aug. 2018, pp. 230-243. [cited by applicant]
Michael Alexander Riegler, EIR—A Medical Multimedia System for Efficient Computer Aided Diagnosis, PhD Thesis, University of Oslo, Oslo, Norway, Feb. 2017, 281 pages. [cited by applicant]
Moeskops et al., Adversarial Training and Dilated Convolutions for Brain MRI Segmentation, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, retrieved fromhttps://link.spring… [cited by applicant]
Mori et al., Computer-aided diagnosis for colonoscopy, Endoscopy, vol. 49, No. 8, Aug. 2017, pp. 813-819. [cited by applicant]
Oliva et al., Prototype system for feature extraction, classification and study of medical images, Expert Systems with Applications, vol. 63, Jul. 2016, pp. 267-283. [cited by applicant]
Omid Haji Maghsoudi, Superpixels Based Segmentation and SVM Based Classification Method to Distinguish Five Diseases from Normal Regions in Wireless Capsule Endoscopy, arXiv 2017, available athttps://arxiv.org/abs/1711.… [cited by applicant]
Pogorelov et al., Efficient disease detection in gastrointestinal videos—global features versus neural networks, Multimedia Tools Applications, vol. 76, Jul. 19, 2017, pp. 22493-22525. [cited by applicant]
Qinghui Liu, Deep Learning Applied to Automatic Polyp Detection in Colonoscopy Images, Master's Thesis, University College of Southeast Norway, Kongsberg, Norway, May 2017, 105 pages. [cited by applicant]
Ribeiro et al., Transfer Learning for Colonic Polyp Classification Using Off-the-Shelf CNN Features, 2016 International Workshop on Computer-Assisted and Robotic Endoscopy, Feb. 2017, pp. 1-13. [cited by applicant]
Ross et al., Exploiting the potential of unlabeled endoscopic video data with self-supervised learning, International Journal of Computer Assisted Radiology and Surgery, vol. 13, No. 6, Apr. 27, 2018, pp. 925-933. [cited by applicant]
Schlegl et al., Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery, arXiv.org, Cornell University Library, Mar. 17, 2017, pp. 1-12. [cited by applicant]
Search Report in European Application No. 18180570.6 dated Feb. 12, 2019, 8 pages. [cited by applicant]
Tajbakhsh et al., Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information, IEEE Transactions on Medical Imaging, vol. 32, No. 2, Feb. 2016, pp. 630-644. [cited by applicant]
Tajbakhsh et al., Convolutional Neural Networks for Medical Image Analysis: Fine Tuning or Full Training?, IEEE Transactions on Medical Imaging, vol. 35, No. 5, May 2016, pp. 1299-1312. [cited by applicant]
Tamaki et al., Computer-aided colorectal tumor classification in NBI endoscopy using local features, Medical Image Analysis, vol. 17, Jan. 2013, pp. 78-100. [cited by applicant]
Vazquez et al., A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images, Journal of Healthcare Engineering, vol. 2017, Published Jul. 26, 2017, 9 pages. [cited by applicant]
Wang et al., Polyp-Alert: Near real-time feedback during colonoscopy, Computer Methods and Programs in Biomedicine, vol. 120, Jul. 2015, pp. 164-179. [cited by applicant]
Wimmer et al., Directional wavelet based features for colonic polyp classification, Medical Image Analysis, vol. 31, Jul. 2016, pp. 16-36. [cited by applicant]
Yu et al., Integrating Online and Offline Three-Dimensional Deep Learning for Automated Polyp Detection in Colonoscopy Videos, IEEE Journal of Biomedical and Health Informatics, vol. 21, No. 1, Jan. 2017, pp. 65-75. [cited by applicant]
Yuan et al., Automatic Polyp Detection in Colonoscopy Videos, Medical Imaging 2017: Image Processing, Feb. 2017, 10 pages. [cited by applicant]
Yuan et al., Deep learning for polyp recognition in wireless capsule endoscopy images, Medical Physics, vol. 44, No. 4, Apr. 2017, pp. 1379-1389. [cited by applicant]
Yuan et al., Improved Bag of Feature for Automatic Polyp Detection in Wireless Capsule Endoscopy Images, IEEE Transactions on Automation Science and Engineering, vol. 13, No. 2, Apr. 2016, pp. 529-535. [cited by applicant]
Zhang et al., Automatic Detection and Classification of Colorectal Polyps by Transferring Low-level CNN Features from Nonmedical Domain, IEEE Journal of Biomedical and Health Informatics, vol. 21, No. 1, Jan. 2017, pp. … [cited by applicant]
Zhu et al., Deep Feature Flow for Video Recognition, arXiv 2017, available at https://arxiv.org/abs/1611.07715, Nov. 2016, 10 pages. [cited by applicant]