Medical image segmentation and severity grading using neural network architectures with semi-supervised learning techniques
This disclosure relates to improved techniques for performing computer vision functions on medical images, including object segmentation functions for identifying medical objects in the medical images and grading functions for determining severity labels for medical conditions exhibited in the medical images. The techniques described herein utilize a neural network architecture to perform these and other functions. The neural network architecture can be trained, at least in part, using semi-supervised learning techniques that enable the neural network architecture to accurately perform the object segmentation and grading functions despite limited availability of pixel-level annotation information.
1. A computer vision system for analyzing medical images comprising:
one or more computing devices comprising one or more processors and one or more non-transitory storage devices for storing instructions, wherein execution of the instructions by the one or more processors causes the one or more computing devices to:
receive a set of training images including a first subset of training images comprising pixel-level annotation information and a second subset of training images comprising image-level annotation information;
execute a training procedure that jointly trains a segmentation model to identify medical objects included in medical images and a grading model to assign severity classification labels to the medical images, wherein executing the training procedure includes:
executing a pre-training procedure that uses a fully-supervised training approach to train the segmentation model with the first subset of training images and the grading model with the second subset of training images;
generating, using the segmentation model, predicted masks based on the second subset of training images after the pre-training procedure is performed;
utilizing the predicted masks to train an attention function of the grading model;
generating, using the attention function of the grading model, pseudo masks based on the second subset of training images; and
utilizing the predicted masks and the pseudo masks to further train the segmentation model using a semi-supervised training approach;
receive a medical image; and
generate, using the segmentation model and the grading model, analysis information for the medical image that identifies a severity classification label for the medical image and one or more medical objects included in the medical image.
2. The system of claim 1 , wherein the segmentation model includes an encoder-decoder structure that is implemented, at least in part, with a fully convolutional network.
3. The system of claim 1 , wherein the grading model is implemented with one or more convolutional neural networks and the grading model comprises:
a grading function that is trained to assign the severity classification labels to the medical images; and
the attention function that generates the pseudo masks which are utilized to train the segmentation model using the semi-supervised training approach.
4. The system of claim 1 , wherein utilizing the pseudo masks to further train the segmentation model using the semi-supervised training approach includes:
generating the pseudo masks using the attention function of the grading model;
providing the predicted masks generated by the segmentation model to a real data branch of a generative adversarial network that includes a discriminator;
providing the pseudo masks generated by the grading model to a fake data branch of a generative adversarial network;
analyzing the predicted masks and pseudo masks with the discriminator of the generative adversarial network, wherein the discriminator aims to distinguish the predicted masks from the pseudo masks; and
utilizing feedback from the discriminator to further train the segmentation model.
5. The system of claim 1 , wherein pre-training the segmentation model using the fully-supervised training approach includes utilizing a binary cross-entropy loss to minimize distances between the predicted masks generated by the segmentation model and ground-truths masks that are included in the pixel-level annotation information associated with the first subset of training images.
6. The system of claim 1 , wherein:
the computer vision system is trained to perform functions associated with diagnosing or assessing a diabetic retinopathy condition;
the medical objects correspond to lesions objects;
the predicted masks correspond to lesion masks that identify locations of the lesion objects;
the segmentation model is trained to generate the lesion masks; and
the grading model is trained to assign the severity classification label to the medical image; and
the severity classification label indicates a severity or stage of the diabetic retinopathy condition associated with the medial image.
7. The system of claim 6 , wherein:
the set of training images includes eye-related images;
the first subset of training images include pixel-level annotation information identifying locations of the lesion objects in the first subset of training images;
the pixel-level annotation information included with the first subset of training images includes ground-truth masks identifying the locations of the lesion objects;
the second subset of training images includes image-level annotation information that includes the severity classification labels corresponding to the diabetic retinopathy condition; and
the second subset of training images is larger than the first subset of training images.
8. The system of claim 6 , wherein:
the segmentation model is capable of generating the lesion masks for a plurality of lesion object types; and
the plurality of lesion object types at least include: micro-aneurysms, hemorrhages, hard exudates, and soft exudates.
9. The system of claim 1 , wherein:
the computer vision system is trained to perform functions associated with diagnosing or assessing one or more of: a cancer-related condition; a bone-related condition; a nerve-related condition; a heart-related condition; an organ-related condition; a blood-related condition; or a brain-related condition.
10. The system of claim 1 , wherein the computer vision system is incorporated into, or communicates with, a system or application that provides medical services.
11. A method for providing a computer vision system that analyzes medical images comprising:
receiving a set of training images including a first subset of training images comprising pixel-level annotation information and a second subset of training images comprising image-level annotation information;
executing a training procedure that jointly trains a segmentation model to identify medical objects included in medical images and a grading model to assign severity classification labels to the medical images, wherein executing the training procedure includes:
executing a pre-training procedure that uses a fully-supervised training approach to train the segmentation model with the first subset of training images and the grading model with the second subset of training images;
generating, using the segmentation model, predicted masks based on the second subset of training images after the pre-training procedure is performed;
utilizing the predicted masks to train an attention function of the grading model;
generating, using the attention function of the grading model, pseudo masks based on the second subset of training images; and
utilizing the predicted masks and the pseudo masks to further train the segmentation model using a semi-supervised training approach;
receiving a medical image; and
generating, using the segmentation model and the grading model, analysis information for the medical image that identifies a severity classification label for the medical image and one or more medical objects included in the medical image.
12. The method of claim 11 , wherein the segmentation model includes an encoder-decoder structure that is implemented, at least in part, with a fully convolutional network.
13. The method of claim 11 , wherein the grading model is implemented with one or more convolutional neural networks and the grading model comprises:
a grading function that is trained to assign the severity classification labels to the medical images; and
the attention function that generates the pseudo masks which are utilized to train the segmentation model using the semi-supervised training approach.
14. The method of claim 11 , wherein utilizing the pseudo masks to further train the segmentation model using the semi-supervised training approach includes:
generating the pseudo masks using the attention function of the grading model;
providing the predicted masks generated by the segmentation model to a real data branch of a generative adversarial network that includes a discriminator;
providing the pseudo masks generated by the grading model to a fake data branch of a generative adversarial network;
analyzing the predicted masks and pseudo masks with the discriminator of the generative adversarial network, wherein the discriminator aims to distinguish the predicted masks from the pseudo masks; and
utilizing feedback from the discriminator to further train the segmentation model.
15. The method of claim 11 , wherein pre-training the segmentation model using the fully-supervised training approach includes utilizing a binary cross-entropy loss to minimize distances between the predicted masks generated by the segmentation model and ground-truths masks that are included in the pixel-level annotation information associated with the first subset of training images.
16. The method of claim 11 , wherein:
the computer vision system is trained to perform functions associated with diagnosing or assessing a diabetic retinopathy condition;
the medical objects correspond to lesions objects;
the predicted masks correspond to lesion masks that identify locations of the lesion objects;
the segmentation model is trained to generate the lesion masks; and
the grading model is trained to assign the severity classification label to the medical image; and
the severity classification label indicates a severity or stage of the diabetic retinopathy condition associated with the medial image.
17. The method of claim 16 , wherein:
the set of training images includes eye-related images;
the first subset of training images include pixel-level annotation information identifying locations of the lesion objects in the first subset of training images;
the pixel-level annotation information included with the first subset of training images includes ground-truth masks identifying the locations of the lesion objects;
the second subset of training images includes image-level annotation information that includes the severity classification labels corresponding to the diabetic retinopathy condition; and
the second subset of training images is larger than the first subset of training images.
18. The method of claim 16 , wherein:
the segmentation model is capable of generating the lesion masks for a plurality of lesion object types; and
the plurality of lesion object types at least include: micro-aneurysms, hemorrhages, hard exudates, and soft exudates.
19. The method of claim 11 , wherein:
the computer vision system is trained to perform functions associated with diagnosing or assessing one or more of: a cancer-related condition; a bone-related condition; a nerve-related condition; a heart-related condition; an organ-related condition; a blood-related condition; or a brain-related condition.
20. A computer program product comprising a non-transitory computer-readable medium including instructions for causing a computer to:
receive a set of training images including a first subset of training images comprising pixel-level annotation information and a second subset of training images comprising image-level annotation information;
execute a training procedure that jointly trains a segmentation model to identify medical objects included in medical images and a grading model to assign severity classification labels to the medical images, wherein executing the training procedure includes:
executing a pre-training procedure that uses a fully-supervised training approach to train the segmentation model with the first subset of training images and the grading model with the second subset of training images;
generating, using the segmentation model, predicted masks based on the second subset of training images after the pre-training procedure is performed;
utilizing the predicted masks to train an attention function of the grading model;
generating, using the attention function of the grading model, pseudo masks based on the second subset of training images; and
utilizing the predicted masks and the pseudo masks to further train the segmentation model using a semi-supervised training approach;
receive a medical image; and
generate, using the segmentation model and the grading model, analysis information for the medical image that identifies a severity classification label for the medical image and one or more medical objects included in the medical image.