Cross-modality neural network transform for semi-automatic medical image annotation
View Patent ↗A cross-modality neural network transform for semi-automatic medical image annotation is provided. In various embodiments, an input medical image is mapped to a first vector in a text vector space. The first vector corresponds to the features of the medical image. A set of predetermined vectors is searched for a closest one of the predetermined vectors to the first vector. From the closest one of the predetermined vectors, one or more keywords is determined describing the input medical image.
1 . A method comprising:
providing an input medical image as input to a trained convolutional neural network, wherein the trained convolutional neural network is configured to generate a feature vector based on the input thereto;
reading the feature vector from the trained convolutional neural network;
determining a first vector from the input medical image via the feature vector, said determining comprising:
providing the feature vector as input to a first trained neural network, wherein the first trained neural network is configured to generate the first vector in a text vector space based on the feature vector, wherein the first vector represents a predicted text report corresponding to features of the input medical image, and
reading the first vector from the first trained neural network;
generating a set of vectors in the text vector space, wherein said generating comprises:
providing a set of text reports as input to a second trained neural network comprising a Doc2Vec model, and
reading from the second trained neural network, for each text report of the set of text reports, a vector representing that text report in the text vector space;
searching the set of vectors in the text vector space to identify a closest of the set of vectors to the first vector; and
from the closest of the set of vectors, determining one or more keywords describing the input medical image.
2 . The method of claim 1 , wherein the neural network is a feedforward neural network.
3 . The method of claim 1 , wherein determining the one or more keywords comprises identifying descriptors in a source text.
4 . The method of claim 1 , wherein the closest of the predetermined vectors is determined by distance within the text vector space of the predetermined vectors to the first vector.
5 . The method of claim 1 , wherein the closest of the predetermined vectors is determined by Mahalanobis distance within the text vector space of the predetermined vectors to the first vector.