System and method for generating and editing diagnosis reports based on medical images
Embodiments of the disclosure provide systems and methods for generating a report based on a medical image of a patient. An exemplary system includes a communication interface configured to receive the medical image acquired by an image acquisition device. The system may further include at least one processor. The at least one processor is configured to automatically determine keywords from a natural language description of the medical image generated by applying a learning network to the medical image. The at least one processor is further configured to generate the report describing the medical image of the patient based on the keywords. The at least one processor is also configured to provide the report for display.
1. A system for generating a report based on a medical image of a patient, comprising:
a communication interface configured to receive the medical image acquired by an image acquisition device; and
at least one processor, configured to:
generate a natural language description of the medical image as a direct output of a learning network applied to the medical image, wherein the learning network is configured to extract image features from the medical image, assign weighs to the image features, and generate the natural language description of the medical image based on the extracted image features and the assigned weights;
automatically determine keywords from the natural language description of the medical image;
generate the report describing the medical image of the patient based on the keywords; and
provide the report for display.
2. The system of claim 1 , wherein the learning network comprises a convolutional neural network and a recursive neural network connected in series, wherein the convolutional neural network is configured to extract the image features from the medical image and the recursive neural network is configured to generate the natural language description of the medical image.
3. The system of claim 2 , wherein the learning network further comprises an attention layer in between the convolutional neural network and the recursive neural network, wherein the attention layer is configured to assign the weights to the image features, wherein the recursive neural network is configured to generate the natural language description of the medical image based on the extracted image features respectively weighted by the weights.
4. The system of claim 2 , wherein the convolutional neural network comprises a VGG16/19 CNN network or a Densenet CNN network.
5. The system of claim 2 , wherein the recursive neural network comprises at least one of a long short-term memory (LSTM) or a gate recurrent unit (GRU).
6. The system of claim 1 , wherein to generate the report based on the keywords, the at least one processor is further configured to:
receive a keyword selection among the keywords; and
generate the report based on the keyword selection.
7. The system of claim 6 , wherein the keyword selection is made by a user through a voice input, an annotation, or a text input.
8. The system of claim 1 , wherein the at least one processor is further configured to:
receive a selection of a region of interest in the medical image; and
automatically update the report to describe the selected region of interest based on the learning network.
9. The system of claim 1 , wherein the at least one processor is further configured to:
receive a selection of a view of the medical image; and
automatically update the report to describe the selected view based on the learning network.
10. The system of claim 1 , further comprising a display configured to display the report with the medical image, wherein the display includes a touchscreen configured to receive a user interaction with the medical image.
11. A method for generating a report based on a medical image of a patient, comprising:
receiving the medical image; by a communication interface, wherein the medical image is acquired by an image acquisition device;
generating, by at least one processor, a natural language description of the medical image as a direct output of a learning network applied to the medical image, wherein the learning network is configured to extract image features from the medical image, assign weighs to the image features, and generate the natural language description of the medical image based on the extracted image features and the assigned weights;
automatically determining, by the at least one processor, keywords from the natural language description of the medical image;
generating the report, by the at least one processor, describing the medical image of the patient based on the keywords; and
providing the report for display.
12. The method of claim 11 , wherein the learning network comprises a convolutional neural network and a recursive neural network connected in series, wherein the convolutional neural network is configured to extract the image features from the medical image and the recursive neural network is configured to generate the natural language description of the medical image.
13. The method of claim 12 , wherein the convolutional neural network comprises a VGG16/19 CNN network or a Densenet CNN network.
14. The method of claim 12 , wherein the recursive neural network comprises at least one of a long short-term memory (LSTM) or a gate recurrent unit (GRU).
15. The method of claim 12 , wherein the learning network further comprises an attention layer in between the convolutional neural network and the recursive neural network, wherein the attention layer is configured to assign the weights to the image features, wherein the recursive neural network is configured to generate the natural language description of the medical image based on the extracted image features respectively weighted by the weights.
16. The method of claim 11 , wherein generating the report based on the keywords further comprising:
receiving a keyword selection among the keywords, wherein the keyword selection is made by a user through a voice input, an annotation, or a text input; and
generating the report based on the keyword selection.
17. The method of claim 11 , further comprising:
receiving a selection of a region of interest in the medical image; and
automatically updating the report to describe the selected region of interest based on the learning network.
18. The method of claim 11 , further comprising:
receiving a selection of a view of the medical image; and
automatically update the report to describe the selected view based on the learning network.
19. A non-transitory computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by at least one processor, performs a method for generating a report based on a medical image of a patient, comprising:
receiving the medical image acquired by an image acquisition device;
generating a natural language description of the medical image as a direct output of a learning network applied to the medical image, wherein the learning network is configured to extract image features from the medical image assign weighs to the image features, and generate the natural language description of the medical image based on the extracted image features and the assigned weights;
automatically determining keywords from the natural language description of the medical image;
generating the report describing the medical image of the patient based on the keywords; and
providing the report for display.
20. The non-transitory computer-readable medium of claim 19 , wherein the learning network comprises a convolutional neural network and a recursive neural network connected in series, wherein the convolutional neural network is configured to extract the image features from the medical image and the recursive neural network is configured to generate the natural language description of the medical image.