Deep neural network framework for processing oct images to predict treatment intensity
Systems and methods relate to processing optical tomography coherence (OCT) images to predict characteristics of a treatment to be administered to effectively treat age-related macular degeneration. The processing can include pre-processing the image by flattening and/or cropping the image and processing the pre-processed image using a neural network. The neural network can include a deep convolutional neural network. An output of the neural network can indicate a predicted frequency and/or interval at which a treatment (e.g., anti-vascular endothelial growth factor therapy) is to be administered so as to prevent leakage of vasculature in the eye.
1. A method comprising:
accessing an optical coherence tomography (OCT) image that corresponds to an eye of a subject experiencing age-related macular degeneration;
identifying, within the OCT image, a set of pixels that correspond to a retina layer;
flattening the OCT image based on the set of pixels;
generating a plurality of patches using the flattened OCT image, wherein generating the plurality of patches comprises:
performing one or more cropping processes using the flattened OCT image to produce one or more cropped images; and
extracting the plurality of patches from the one or more cropped images;
wherein the plurality of patches comprises patches having a plurality of sizes;
inputting the plurality of patches into a plurality of a patch-specific neural networks;
wherein each patch-specific neural network has been trained, on training patches having a specific size, to predict an effective characteristic of a treatment schedule; and
wherein the plurality of patches is input, based on the size of each patch, to the plurality of patch-specific neural networks;
generating, by the plurality of a patch-specific neural networks, a plurality of patch-specific outputs;
wherein each plurality of patch-specific outputs corresponds to the respective one of the plurality of patches; and
wherein each output of the plurality of patch-specific outputs predicts an effective characteristic of a proposed treatment schedule for the eye of the subject;
weighting, by an integrating neural network that has learned a weighting relationship, the plurality of patch-specific outputs; and
generating, by the integrating neural network and based on the weighted plurality of patch-specific outputs, a label corresponding to the characteristic of the proposed treatment schedule for the eye of the subject;
and
outputting the label.
2. The method of claim 1 ,
wherein outputting the label comprises outputting the label either at or to a client device for use in administering an anti-vascular endothelial growth factor (VEGF) treatment to the eye of the subject according to the proposed treatment schedule; and
wherein the label is indicative of a frequency of treatment administration predicted to be sufficiently effective such that fluid does not leak from vessels in the eye between successive treatment administration.
3. The method of claim 1 , wherein the characteristic of the proposed treatment schedule indicates an interval between successive administrations of a treatment.
4. The method of claim 1 , wherein the retina layer includes a retina pigment epithelium layer.
5. The method of claim 1 ,
wherein the label identifies the eye of the subject as an eye predicted to be effectively treated by the characteristic of the proposed treatment schedule and based on the weighted plurality of patch-specific outputs that is generated by the integrating neural network;
wherein the characteristic of the proposed treatment schedule comprises an interval between successive administrations of an anti-vascular endothelial growth factor (aVEGF) treatment; and
wherein the method further comprises administering the aVEGF treatment to the eye of the subject according to the proposed treatment schedule.
6. The method of claim 1 , wherein the weighting relationship comprises applying a weight to a patch-specific output based on the patch-specific neural network that generated the patch-specific output.
7. The method of claim 1 , wherein the weighting relationship comprises applying a weight to each patch-specific output based on the other path-specific outputs.
8. A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of actions including:
accessing an optical coherence tomography (OCT) image that corresponds to an eye of a subject experiencing age-related macular degeneration;
identifying, within the OCT image, a set of pixels that correspond to a retina layer;
flattening the OCT image based on the set of pixels;
generating a plurality of patches using the flattened OCT image, wherein generating the plurality of patches comprises:
performing one or more cropping processes using the flattened OCT image to produce one or more cropped images; and
extracting the plurality of patches from the one or more cropped images;
wherein the plurality of patches comprises patches having a plurality of sizes;
inputting the plurality of patches into a plurality of a patch-specific neural networks;
wherein each patch-specific neural network has been trained, on training patches having a specific size, to predict an effective characteristic of a treatment schedule; and
wherein the plurality of patches is input, based on the size of each patch, to the plurality of patch-specific neural networks;
generating, by the plurality of a patch-specific neural networks, a plurality of patch-specific outputs;
wherein each plurality of patch-specific outputs corresponds to the respective one of the plurality of patches; and
wherein each output of the plurality of patch-specific outputs predicts an effective characteristic of a proposed treatment schedule for the eye of the subject;
weighting, by an integrating neural network that has learned a weighting relationship, the plurality of patch-specific outputs; and
generating, by the integrating neural network and based on the weighted plurality of patch-specific outputs, a label corresponding to the characteristic of the proposed treatment schedule for the eye of the subject;
and
outputting the label.
9. The system of claim 8 ,
wherein outputting the label comprises outputting the label either at or to a client device for use in administering an anti-vascular endothelial growth factor (VEGF) treatment to the eye of the subject according to the proposed treatment schedule; and
wherein the label is indicative of a frequency of treatment administrations predicted to be sufficiently effective such that fluid does not leak from vessels in the eye between successive treatment administrations.
10. The system of claim 8 , wherein the characteristic of the proposed treatment schedule indicates an interval between successive administrations of a treatment.
11. The system of claim 8 , wherein the retina layer includes a retina pigment epithelium layer.
12. The system of claim 8 , wherein the proposed treatment schedule includes a proposed schedule for administering anti-vascular endothelial growth factor.
13. The system of claim 8 , wherein the weighting relationship comprises applying a weight to a patch-specific output based on the patch-specific neural network that generated the patch-specific output.
14. The system of claim 8 , wherein the weighting relationship comprises applying a weight to each patch-specific output based on the other path-specific outputs.
15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a set of actions including:
accessing an optical coherence tomography (OCT) image that corresponds to an eye of a subject experiencing age-related macular degeneration;
identifying, within the OCT image, a set of pixels that correspond to a retina layer;
flattening the OCT image based on the set of pixels;
generating a plurality of patches using the flattened OCT image, wherein generating the plurality of patches comprises:
performing one or more cropping processes using the flattened OCT image to produce one or more cropped images; and
extracting the plurality of patches from the one or more cropped images;
wherein the plurality of patches comprises patches having a plurality of sizes;
inputting the plurality of patches into a plurality of a patch-specific neural networks;
wherein each patch-specific neural network has been trained, on training patches having a specific size, to predict an effective characteristic of a treatment schedule; and
wherein the plurality of patches is input, based on the size of each patch, to the plurality of patch-specific neural networks;
generating, by the plurality of a patch-specific neural networks, a plurality of patch-specific outputs;
wherein each plurality of patch-specific outputs corresponds to the respective one of the plurality of patches; and
wherein each output of the plurality of patch-specific outputs predicts an effective characteristic of a proposed treatment schedule for the eye of the subject;
weighting, by an integrating neural network that has learned a weighting relationship, the plurality of patch-specific outputs; and
generating, by the integrating neural network and based on the weighted plurality of patch-specific outputs, a label corresponding to the characteristic of the proposed treatment schedule for the eye of the subject; and
outputting the label.
16. The computer-program product of claim 15 ,
wherein outputting the label comprises outputting the label either at or to a client device for use in administering an anti-vascular endothelial growth factor (VEGF) treatment to the eye of the subject according to the proposed treatment schedule; and
wherein the label is indicative of a frequency of treatment administrations or an interval between successive treatment administrations predicted to be sufficiently effective such that fluid does not leak from vessels in the eye between successive treatment administrations.
17. The computer-program product of claim 15 , wherein the weighting relationship comprises applying a weight to a patch-specific output based on the patch-specific neural network that generated the patch-specific output.
18. The computer-program product of claim 15 , wherein the weighting relationship comprises applying a weight to each patch-specific output based on the other path-specific outputs.