IP Library Granted Patent US 11,971,960
Granted Patent B2
US 11,971,960 · App. 17/309,100 · Granted Apr 30, 2024

Deep learning based image enhancement

Inventors: Nicholas J. Durr (Baltimore, MD); Taylor L Bobrow (Chesapeake, VA); Faisal Mahmood (Baltimore, MD)
Assignee: The Johns Hopkins University
G06F18/24G06N3/045G06N3/088G16H30/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,971,960
App. No.
17/309,100
Granted
Apr 30, 2024
Kind
B2
Abstract

A device may receive a coherent energy illuminated image, of a particular object, that includes laser speckle. The device may process, using a laser speckle reduction model, the coherent energy illuminated image to generate a laser speckle-reduced image. The device may provide the laser speckle-reduced image as output to permit diagnostics based on the laser speckle-reduced image.

Claims (57)

1. A device, comprising:

one or more memories; and

one or more processors, communicatively coupled to the one or more memories, to:

receive a training data set for training a laser speckle reduction model,

wherein the training data set includes a set of image pairs of training objects, and

wherein an image pair, of the set of image pairs, includes a first image, of a training object of the training objects, that includes laser speckle and a second image, of the training object, that includes less laser speckle than the first image;

train, using a deep learning technique, the laser speckle reduction model based on the training data set,

wherein the laser speckle reduction model comprises a generator model to generate a reduced laser speckle image and a discriminator model to train the generator model;

receive, after training the laser speckle reduction model, a coherent energy illuminated image, of a particular object not included in the training objects, that includes laser speckle;

process, using the laser speckle reduction model, the coherent energy illuminated image to generate a laser speckle-reduced image,

wherein processing the coherent energy illuminated image comprises processing the coherent energy illuminated image in connection with an incoherent energy illuminated image; and

provide an output connected with the laser speckle-reduced image and the incoherent energy illuminated image.

2. The device of claim 1 , wherein the one or more processors are further to:

perform image processing on the laser speckle-reduced image to identify a characteristic of the laser speckle-reduced image; and

provide output associated with the characteristic of the laser speckle-reduced image.

3. The device of claim 2 , wherein the particular object is a medical object and the characteristic is a diagnostic characteristic of the laser speckle-reduced image.

4. The device of claim 1 , wherein the coherent energy illuminated image is received from an endoscopic imaging system.

5. The device of claim 1 , wherein the generator model is a trained neural network.

6. The device of claim 1 , wherein the coherent energy illuminated image is altered using one or more additional laser speckle-reduction techniques before the processing of the coherent energy illuminated image.

7. The device of claim 1 , wherein the one or more processors, when training the laser speckle reduction model, are to:

segment the image pair into a set of image segment pairs,

wherein an image segment pair, of the set of image segment pairs, includes a first image segment that includes laser speckle and a corresponding second image segment that does not include laser speckle; and

train the laser speckle reduction model using the set of image segment pairs.

8. The device of claim 1 , wherein the generator model and the discriminator model are conditional generative adversarial networks.

9. The device of claim 1 , wherein the coherent energy illuminated image is illuminated using a plurality of channels of illumination, and

wherein the plurality of channels includes at least one of:

a red channel,

a green channel,

a blue channel, or

a near-infrared channel.

10. The device of claim 1 , wherein the coherent energy illuminated image is obtained for an endoscopy procedure.

11. The device of claim 1 , wherein the device is an endoscope.

12. The device of claim 1 , wherein coherent energy is directed from a laser source fixed at a distal end of an endoscope.

13. The device of claim 1 , wherein coherent energy is directed to the particular object from a laser source via a light guide.

14. A non-transitory computer-readable medium storing instructions, the instructions comprising:

one or more instructions that, when executed by one or more processors, cause the one or more processors to:

receive a training data set for training a laser noise reduction model,

wherein the training data set includes a set of image pairs of training objects, and

wherein an image pair, of the set of image pairs, includes a first image, of a training object of the training objects, that includes laser noise and a second image, of the training object, that includes less laser noise than the first image;

train, using a deep learning technique, the laser noise reduction model based on the training data set,

wherein the laser noise reduction model comprises a generator model to generate a reduced laser noise image and a discriminator model to train the generator model;

provide the laser noise reduction model based on training the laser noise reduction model;

receive, after training the laser noise reduction model, a coherent energy illuminated image, of a particular object not included in the training data set, that includes laser noise;

process, using the laser noise reduction model, the coherent energy illuminated image to generate a laser noise-reduced image;

provide the laser noise-reduced image as output;

perform image processing on the laser noise-reduced image to identify a characteristic of the laser noise-reduced image; and

provide additional output associated with the characteristic of the laser noise-reduced image.

15. The non-transitory computer-readable medium of claim 14 , wherein the laser noise-reduced image is associated with a greater than 3 decibel reduction in noise relative to the coherent energy illuminated image.

16. The non-transitory computer-readable medium of claim 14 , wherein the deep learning technique is a conditional generative adversarial networks neural network technique.

17. A method, comprising:

receiving, by a device, a coherent energy illuminated image, of a particular object, that includes laser speckle;

processing, by the device and using a laser speckle reduction model, the coherent energy illuminated image to generate a laser speckle-reduced image,

wherein processing the coherent energy illuminated imagea comprises processing the coherent energy illuminated image in connection with an incoherent energy illuminated image; and

providing, by the device, an output connected with the laser speckle-reduced image and the incoherent energy illuminated image to permit diagnostics based on the laser speckle-reduced image.

18. The method of claim 17 , wherein the laser speckle-reduced image is associated with a greater than 5 decibel reduction in noise relative to the coherent energy illuminated image.

19. The method of claim 17 , wherein the coherent energy illuminated image is received from an endoscopic imaging system.

20. The method of claim 17 , wherein the particular object is a medical object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 1, 2023
From: DURR, NICHOLAS J.; BOBROW, TAYLOR L.; MAHMOOD, FAISAL
To: THE JOHNS HOPKINS UNIVERSITY
Reel/Frame 065733/0936 →
Continuity (2)
Provisional Application 62749242 · Oct 23, 2018
Related Publication 20220019861A1 · Jan 20, 2022