Magnification of medical images with super-resolution
The current disclosure provides methods and systems to provide a magnification capability to a software application for reviewing medical images, where a selected portion of a medical image displayed by the software application can be viewed within a magnification window at a higher resolution than other portions of the medical image. A higher-resolution medical image of the selected portion can be generated from the medical image using a trained convolutional neural network (CNN). In a first embodiment, the selected portion of the medical image is cropped and inputted into the trained CNN to generate a higher-resolution medical image corresponding to the selected portion. In a second embodiment, the medical image is inputted into the trained CNN to generate a higher-resolution medical image when the medical image is displayed, and the selected portion of the higher-resolution medical image is cropped and displayed at a time of selection.
1 . A computer-implemented method for a medical imaging system, the computer-implemented method comprising:
while displaying a first two-dimensional (2-D) medical image having a first resolution in a medical review software application running on a client device of the medical imaging system, the first 2-D medical image generated from an image volume acquired via the medical imaging system, displaying a selected portion of the first 2-D medical image in a second resolution, the second resolution higher than the first resolution, the selected portion selected by a user of the client device,
wherein the selected portion of the first 2-D medical image includes an area defined by a bounding box generated on the first 2-D medical image by the user via an input device of the client device,
wherein displaying the selected portion of the first 2-D medical image in the second resolution further comprises displaying a second 2-D medical image having the second resolution in a magnification window superimposed on the 2-D medical image, the second 2-D medical image generated by a neural network trained to generate a higher-resolution version of 2-D medical images,
wherein the trained neural network is a convolutional neural network (CNN) with an encoder-decoder architecture including an encoder portion and a decoder portion,
the computer-implemented method further comprising inputting a latent space representation of the image volume into the trained neural network as an additional input to generate the second 2-D medical image.
2 . The computer-implemented method of claim 1 , wherein displaying the selected portion of the first 2-D medical image in the second resolution further comprises:
cropping the first 2-D medical image based on the bounding box; and
inputting the cropped first 2-D medical image into the trained neural network to generate the second 2-D medical image.
3 . The computer-implemented method of claim 1 , wherein displaying the selected portion of the first 2-D medical image in the second resolution further comprises:
inputting the first 2-D medical image into the trained neural network to generate the second 2-D medical image;
cropping the second 2-D medical image based on the bounding box; and
displaying the cropped second 2-D medical image in the magnification window.
4 . The computer-implemented method of claim 1 , further comprising:
requesting and receiving the first 2-D medical image from a server of a computing device of the medical imaging system at the client device; and
generating the second 2-D medical image at the client device based on the first 2-D medical image using the trained neural network.
5 . The computer-implemented method of claim 1 , wherein the latent space representation includes a three-dimensional (3-D) portion of a latent space of the image volume corresponding to the first 2-D medical image, and the 3-D portion is inputted into the decoder portion of the trained neural network.
6 . The computer-implemented method of claim 1 , wherein the latent space representation includes a plurality of 2-D images extracted from a latent space of the image volume, each 2-D image of the plurality of 2-D images corresponding to a different channel of the latent space, and the first 2-D medical image is inputted into a first encoder portion of the trained neural network, and the plurality of 2-D images are inputted into a second encoder portion of the trained neural network, the second encoder portion different from the first encoder portion.
7 . The computer-implemented method of claim 6 , wherein the 2-D images extracted from the latent space of the image volume have a same point of view as the first 2-D medical image.
8 . The computer-implemented method of claim 6 , wherein a first output of the first encoder portion and a second output of the second encoder portion are concatenated and inputted into the decoder portion.
9 . A medical imaging system, comprising:
a computing device storing an image volume of a subject of the medical imaging system; and
a client device running a medical review software application, the client device including a processor communicably coupled to a non-transitory memory of the client device including instructions that when executed, cause the processor to:
request a two-dimensional (2-D) medical image of the image volume and a latent space representation of the image volume from the computing device;
display the requested 2-D medical image on a screen of the client device; and
in response to a user of the client device selecting a magnification tool of the medical review software application:
receive a selected portion of the requested 2-D medical image selected by the user;
input the selected portion and the latent space representation of the image volume into a trained convolutional neural network (CNN) stored on the client device;
receive, as an output of the trained CNN, a higher-resolution 2-D medical image corresponding to the selected portion; and
display the higher-resolution 2-D medical image in a magnification window of the medical review software application.
10 . The medical imaging system of claim 9 , wherein the selected portion is inputted into an input layer of an encoder portion of the trained CNN, and the latent space representation is inputted into a first layer of a decoder portion of the trained CNN.
11 . The medical imaging system of claim 9 , wherein the selected portion is inputted into an input layer of a first encoder portion of the trained CNN, the latent space representation is inputted into a second encoder portion of the trained CNN, and a first output of the first encoder portion and a second output of the second encoder portion are concatenated and inputted into a decoder portion of the trained CNN.
12 . The medical imaging system of claim 9 , wherein the selected portion is defined by a bounding box created by the user on the screen, and further instructions are stored in the non-transitory memory that when executed, cause the processor to:
crop the requested 2-D medical image based on the bounding box;
input the cropped 2-D medical image into the trained CNN to generate the higher-resolution 2-D medical image; and
display the cropped 2-D medical image in the magnification window.
13 . The medical imaging system of claim 9 , wherein the selected portion is defined by a bounding box created by the user on the screen, and further instructions are stored in the non-transitory memory that when executed, cause the processor to:
input the requested 2-D medical image into the trained CNN to generate the higher-resolution 2-D medical image;
crop the higher-resolution 2-D medical image based on the bounding box; and
display the cropped higher-resolution 2-D medical image in the magnification window.
14 . A method for a medical imaging system, comprising:
generating a plurality of 2-D medical images from one or more reference image volumes stored in a memory of the medical imaging system;
for each 2-D medical image of the plurality of 2-D medical images, generating a lower-resolution version of the 2-D medical image;
creating a respective plurality of image pairs, each image pair including a 2-D medical image of the plurality of 2-D medical images as a target, ground truth image, and a corresponding lower-resolution version of the 2-D medical image as an input image;
associating with each image pair a latent space representation of a reference image volume corresponding to the image pair;
training a convolutional neural network (CNN) on the image pairs and the latent space representation to increase a resolution of a 2-D medical image using the image pairs; and
deploying the trained CNN at a client device of the medical imaging system, the trained CNN configured to receive as input a 2-D medical image of an image volume acquired from a subject of the medical imaging system and a latent space representation of the image volume, and output a higher-resolution version of the 2-D medical image, the higher-resolution version displayed in a magnification window of a medical review software application running on the client device.
15 . The method of claim 14 , wherein the trained CNN is further configured to input the 2-D medical image into a first encoder portion of the CNN, and input the latent space representation of the image volume into a second encoder portion of the trained CNN.
16 . The method of claim 14 , wherein the trained CNN is further configured to input the latent space representation of the image volume into a decoder portion of the trained CNN.