System for OCT image translation, ophthalmic image denoising, and neural network therefor
An OCT system includes a machine learning (ML) model trained to receive a single OCT scan/image and provide an image translation and/or denoise function. The ML model may be based on a neural network (NN) architecture including a series of encoding modules in a contracting path followed by a series of decoding modules in an expanding path leading to an output convolution module. An intermediate error module determines a deep error measure, e.g., between a training output image and at least one encoding module and/or decoding module, and an error from the output convolution module is combined with the deep error measure. The NN may be trained using true averaged images as ground truth, training outputs. Alternatively, the NN may be trained using randomly selected, individual OCT images/scans as training outputs.
1. An optical coherence tomography (OCT) system comprising:
a light source for generating a beam of light;
a beam splitter having a beam-splitting surface for directing a first portion of the light into a reference arm and a second portion of the light into a sample arm;
optics for directing the light in the sample arm to one or more locations on a sample;
a detector for receiving light returning from the sample and reference arms and generating signals in response thereto;
a processor for converting the signals into a first image and submitting the first image to an image translation module that translates the first image to a second image characterized by one or more of decreased noise artifacts, jitter and minimized creation of fictional structures as compared to the first image; and
an output display for displaying a system output image based on the second image;
wherein the image translation module includes a machine learning module based on a neural network trained using a set of training input images and a target set of training output images,
wherein the neural network has an input module configured to receive a current training input image, a plurality of intermediate processing modules following the input module, an intermediate error module that determines an intermediate error based on an output of at least one select intermediate processing module and a current training output image, and an output module following the plurality of intermediate processing modules that determines a preliminary output error based on its output and the current training output image, and
wherein a total loss error for a current training cycle is defined based on a combination of the preliminary output error and the intermediate error.
2. The system of claim 1 , wherein the processor further combines a current second image with one or more previously obtained second images to generate the system output image.
3. The system of claim 2 , wherein the current second image is combined with the one or more previously obtained second images by one of direct averaging or weighted averaging with second images of higher image quality being weighted more heavily.
4. The system of claim 1 , wherein:
the processor defines a plurality of the first images, submits the plurality first images to the image translation module to produce a corresponding plurality of second images, and calculates motion contrast information from the plurality of second images using an OCT angiography (OCTA) processing technique; and
the system output image displays the motion contrast information.
5. The system of claim 1 , wherein the processor:
defines a plurality of the first images;
submits the plurality of first images to the image translation module to produce a corresponding plurality of second images;
applies an image registration technique to the plurality of second images to produce image alignment settings; and
aligns the plurality of first images based at least in part on the image alignment settings of the plurality of second images.
6. The system of claim 1 , wherein:
the first image is divided into a plurality of first image segments; and
the image translation module individually translates each first image segment into a corresponding second image segment, and combines the second image segments to construct the second image.
7. The system of claim 1 , wherein:
at least one of the training output images is defined as the averaging of a set of OCT test images of the same region of a test sample; and
at least a fraction of the training input images is included in the set of OCT test images.
8. The system of claim 1 , wherein:
the first image is of a first region of the sample; and
the second image has characteristics defined as the averaging of multiple hypothetical OCT scans of the first region with the first image.
9. The system of claim 1 , wherein the training of the neural network includes:
collecting a plurality of OCT test images of a target ophthalmic region;
averaging the plurality of OCT test images to define a corresponding averaged-image of the target ophthalmic region;
separately and individually inputting the OCT test images of the target ophthalmic region as training input images to the neural network, and providing their corresponding averaged-image as their individually corresponding training output image for the neural network.
10. The system of claim 9 , wherein the training of the neural network further includes:
dividing each OCT test image into a plurality of test segments;
dividing their corresponding averaged-image into a plurality of corresponding ground truth segments;
correlating test segments to corresponding ground truth segments;
separately and individually submitting the correlated test segments to the neural network as training input images and providing their correlated ground truth segments as training output images for the neural network.
11. The system of claim 9 , further including combining a currently inputted OCT test image with a corresponding current output of the neural network to define a combination network output, and comparing the combination network output with the corresponding training output image to determine the total loss error for a current training cycle.
12. The system of claim 1 , wherein the training input images and training output images include a mixture of images of healthy eyes and images of diseased eyes.
13. The system of claim 1 , wherein:
the first image is of a first imaging modality; and
the second image simulates a second imaging modality different than the first imaging modality.
14. The system of claim 13 , wherein the first and second modalities are a mixture including one or more of time domain OCT, spectral domain OCT, swept source OCT, and adaptive optics OCT (AO-OCT).
15. The system of claim 1 , wherein:
the OCT system is of a first modality;
the machine learning module is trained using third images taken with a first OCT device of the first modality as the set of training input images and fourth images taken with a second OCT device of a second modality as the target set of training output images, the second modality being different than the first modality; and
the second image has features characteristic of an image generated by an OCT system of the second modality.
16. The system of claim 15 , wherein:
the first OCT device of the first modality is of a non-adaptive optics OCT type;
the second OCT device of the second modality is of an adaptive optics OCT type;
the third images obtained by the first OCT device are bigger than the fourth images obtained by the second OCT device, the third images are divided into third image segments of similar size as the fourth images and each third image segment is correlated to a corresponding fourth image; and
the correlated third segments are separately and individually submitted to the neural network as training input images and their correspondingly correlated fourth images are provided as training output images for the neural network.
17. The system of claim 1 , wherein
the plurality of intermediate processing modules form a contracting path and an expanding path, the contracting path following the input module and including a plurality of encoding modules, each encoding module having a convolution stage, an activation function, and a max pooling operation,
the expanding path following the contracting path and having a plurality of decoding modules, each decoding module concatenates its current value with that of a corresponding encoding module;
the output module is an output convolution module excluding a pooling layer and a sigmoid layer activation function, the output convolution module receiving the output from the last decoding module in the expanding path and producing the preliminary output error; and
the intermediate error module determines the intermediate error for at least one encoding module and/or one decoding module.
18. The system of claim 17 , wherein during training of the neural network:
the intermediate error module determines the error measure as an error between the current training output image and the current value of the intermediate error module's corresponding encoding module and/or decoding module; and
the preliminary output error from the output convolution module is based on the current training output image and the current value of the output convolution module.
19. An ophthalmic imaging system comprising:
a processor for acquiring a first image, and submitting the first image to an image modification module that defines a second image based on the first image; and
an output display for displaying an output image based on the second image;
wherein the image modification module includes a machine model based on a neural network trained using a set of training input images and a target set of training output images, the neural network having:
a) an input module for receiving a current training input image;
b) a contracting path following the input module, the contracting path including a plurality of encoding modules, each encoding module having a convolution stage, an activation function, and a max pooling operation;
c) an expanding path following the contracting path, the expanding path having a plurality of decoding modules, each decoding module concatenating its current value with that of a corresponding encoding module;
d) an output convolution module excluding a pooling layer and an activation function, the output convolution module receiving the output from the last decoding module in the expanding path and producing a preliminary output error; and
e) an intermediate error module that determines an error measure for at least one encoding module and/or one decoding module; and
during training of the neural network, the preliminary output error of the output convolution module is combined with the output error of the intermediate error module.
20. The system of claim 19 , wherein the activation function is a rectifier linear unit or sigmoid layer.
21. The system of claim 19 , wherein the preliminary output error is based on a current training output image and the error measure is based on a current output of the at least one encoding module and/or one decoding module and the current training output image.
22. The system of claim 21 , wherein
a training cycle error for a current training cycle is based on the combined errors from the output convolution module and the intermediate error module.
23. The system of claim 19 , wherein the error measure is based on a square loss function.
24. The system of claim 19 , wherein the ophthalmic imaging system is an optical coherence tomography (OCT) angiography system, and the first image is a vasculature image.
25. The system of claim 19 , wherein the ophthalmic imaging system is an optical coherence tomography (OCT) system of a first modality, and training of the neural network includes:
collecting one or more of third images of different target ophthalmic regions using a first OCT system of the first modality;
collecting one or more of fourth images of the same target ophthalmic regions using a second OCT system of a second modality different than the first modality;
defining one or more of the training output images from the one or more fourth images; and
defining one or more of the training input images from the one or more third images, wherein each input training image has a corresponding training output image.
26. The system of claim 25 , wherein:
the first OCT system of a non-adaptive optics OCT type; and
the second OCT system is of an adaptive optics OCT type.
27. The system of claim 19 , wherein one of the set of training input images or target set of training output images are obtained by an optical coherence tomography (OCT) system, and the other of the set of training input images or target set of training output images are obtained by a fundus imaging system.