Medical image segmentation using an integrated edge guidance module and object segmentation network
This disclosure relates to improved techniques for performing image segmentation functions using neural network architectures. The neural network architecture integrates an edge guidance module and object segmentation network into a single framework for detecting target objects and performing segmentation functions. The neural network architecture can be trained to generate edge-attention representations that preserve the edge information included in images. The neural network architecture can be trained to generate multi-scale feature information that preserves and enhances object-level feature information included in images. The edge-attention representations and multi-scale feature information can be fused to generate segmentation results that identify target object boundaries with increased accuracy.
1. A system for performing image segmentation on one or more images, comprising:
one or more computing devices comprising one or more processors and one or more non-transitory storage devices for storing instructions, wherein execution of the instructions by the one or more processors causes the one or more computing devices to:
receive an image comprising one or more objects at a neural network architecture comprising an encoder-decoder network, wherein the encoder-decoder network includes a plurality of encoding blocks and a plurality of decoding blocks;
generate, using the neural network architecture, an edge-attention representation associated with the image, at least in part, utilizing encoding outputs of one or more encoding blocks included in the encoder-decoder network, wherein the edge-attention representation preserves local edge information included in the image;
generate, using the neural network architecture, multi-scale feature information associated with the image, at least in part, utilizing decoding outputs of one or more decoding blocks included in the encoder-decoder network, wherein the multi-scale feature information preserves object-level features included in the image;
generate, using the neural network architecture, segmentation results corresponding to the image using both the edge-attention representation and the multi-scale feature information derived from the image, wherein the segmentation results identify boundaries of the one or more objects in the image.
2. The system of claim 1 , wherein generating the segmentation results using both the edge-attention representation and the multi-scale feature information includes:
concatenating the edge-attention representation with the multi-scale feature information; and
utilizing a convolutional layer to extract the one or more objects included in the image.
3. The system of claim 1 , wherein generating the multi-scale feature information includes:
generating, at each of the plurality of decoding blocks, the decoding outputs;
providing the decoding outputs to weighted blocks that apply weights to identify relevant object-level features; and
aggregating outputs from the weighted blocks to generate the multi-scale feature information.
4. The system of claim 1 , wherein generating the edge-attention representation includes:
generating, using a first encoding block of the encoder-decoder network, a first encoding output having a first resolution;
generating, using a second encoding block of the encoder-decoder network, a second encoding output having a second resolution;
upsampling the second encoding output to the first resolution;
applying one or more convolutional functions to both the first encoding output and the upsampled second encoding output; and
after applying the one or more convolutional functions, generating the edge-attention representation by concatenating the first encoding output and the upsampled second encoding output.
5. The system of claim 1 , wherein the neural network architecture is trained, at least in part, using a segmentation loss function that is utilized to optimize generation of the edge-attention representation and the multi-scale feature information.
6. The system of claim 5 , wherein the loss function utilizes a separate Lovász-Softmax loss to optimize generation of the edge-attention representation and the multi-scale feature information.
7. The system of claim 5 , wherein the neural network architecture is trained using a set of training images that are annotated with pixel-level labels identifying boundaries of objects included in the training images.
8. The system of claim 1 , wherein:
the image is a medical image that includes one or more medical objects;
the one or more medical objects include at least one of: an optic disk, an optic cup, an eye vessel, or a lung; and
the segmentation results identify the boundaries of the one or more medical objects with pixel-level accuracy.
9. The system of claim 1 , wherein:
the encoder-decoder network comprises four encoding blocks and three decoding blocks;
the encoding outputs from a first and second encoding block are utilized to generate the edge-attention representation;
the decoding outputs from each of the three decoding blocks are utilized to generate the multi-scale feature information.
10. The system of claim 1 , wherein:
the system comprises a network;
the image is transmitted to the neural network architecture over the network;
the segmentation results generated by the neural network architecture are transmitted to at least one computing device over the network.
11. A method for performing image segmentation on one or more images, comprising:
receiving an image comprising one or more objects at a neural network architecture comprising an encoder-decoder network, wherein the encoder-decoder network includes a plurality of encoding blocks and a plurality of decoding blocks;
generating, using the neural network architecture, an edge-attention representation associated with the image, at least in part, utilizing encoding outputs of one or more encoding blocks included in the encoder-decoder network, wherein the edge-attention representation preserves local edge information included in the image;
generating, using the neural network architecture, multi-scale feature information associated with the image, at least in part, utilizing decoding outputs of one or more decoding blocks included in the encoder-decoder network, wherein the multi-scale feature information preserves object-level features included in the image;
generating, using the neural network architecture, segmentation results corresponding to the image using both the edge-attention representation and the multi-scale feature information derived from the image, wherein the segmentation results identify boundaries of the one or more objects in the image.
12. The method of claim 11 , wherein generating the segmentation results using both the edge-attention representation and the multi-scale feature information includes:
concatenating the edge-attention representation with the multi-scale feature information; and
utilizing a convolutional layer to extract the one or more objects included in the image.
13. The method of claim 11 , wherein generating the multi-scale feature information includes:
generating, at each of the plurality of decoding blocks, the decoding outputs;
providing the decoding outputs to weighted blocks that apply weights to identify relevant object-level features; and
aggregating outputs from the weighted blocks to generate the multi-scale feature information.
14. The method of claim 11 , wherein generating the edge-attention representation includes:
generating, using a first encoding block of the encoder-decoder network, a first encoding output having a first resolution;
generating, using a second encoding block of the encoder-decoder network, a second encoding output having a second resolution;
upsampling the second encoding output to the first resolution;
applying one or more convolutional functions to both the first encoding output and the upsampled second encoding output; and
after applying the one or more convolutional functions, generating the edge-attention representation by concatenating the first encoding output and the upsampled second encoding output.
15. The method of claim 11 , wherein the neural network architecture is trained, at least in part, using a segmentation loss function that is utilized to optimize generation of the edge-attention representation and the multi-scale feature information.
16. The method of claim 15 , wherein the loss function utilizes a separate Lovász-Softmax loss to optimize generation of the edge-attention representation and the multi-scale feature information.
17. The method of claim 15 , wherein the neural network architecture is trained using a set of training images that are annotated with pixel-level labels identifying boundaries of objects included in the training images.
18. The method of claim 11 , wherein:
the image is a medical image that includes one or more medical objects;
the one or more medical objects include at least one of: an optic disk, an optic cup, an eye vessel, or a lung; and
the segmentation results identify the boundaries of the one or more medical objects with pixel-level accuracy.
19. The method of claim 11 , wherein:
the encoder-decoder network comprises four encoding blocks and three decoding blocks;
the encoding outputs from a first and second encoding block are utilized to generate the edge-attention representation;
the decoding outputs from each of the three decoding blocks are utilized to generate the multi-scale feature information.
20. A computer program product comprising a non-transitory computer-readable medium, including instructions for causing a computer to:
receive an image comprising one or more objects at a neural network architecture comprising an encoder-decoder network, wherein the encoder-decoder network includes a plurality of encoding blocks and a plurality of decoding blocks;
generate, using the neural network architecture, an edge-attention representation associated with the image, at least in part, utilizing encoding outputs of one or more encoding blocks included in the encoder-decoder network, wherein the edge-attention representation preserves local edge information included in the image;
generate, using the neural network architecture, multi-scale feature information associated with the image, at least in part, utilizing decoding outputs of one or more decoding blocks included in the encoder-decoder network, wherein the multi-scale feature information preserves object-level features included in the image;
generate, using the neural network architecture, segmentation results corresponding to the image using both the edge-attention representation and the multi-scale feature information derived from the image, wherein the segmentation results identify boundaries of the one or more objects in the image.