Method and network to embed image data and meta data
The present invention relates to a system and method to embed meta data from an imaging and communications system whereby the meta data is combined with image data as an input to a deep learning network. An image classification learning network is disclosed which comprises: a means to input image data and meta data; and an embedding layer comprising learnable embedding weights to encode the meta data to provide a learned object, and a softmax layer to classify a combination of the image data and the learned object.
1 . An image classification learning network comprising:
an embedding layer comprising learnable embedding weights to encode meta data to provide a learned object; and
a softmax layer to classify a combination of image data and the learned object, wherein the embedding layer comprises a multiplication of a meta data vector comprising the meta data multiplying a weights matrix comprising the learnable embedding weights to create an embedding matrix, and wherein the weights matrix comprises one row of the learnable embedding weights to weight all elements of the meta data equally, wherein the image data and the meta data are obtained from a standardized transmission or storage.
2 . The image classification learning network according to claim 1 , wherein the embedding layer comprises a fully connected layer of neurons to regress an embedding matrix comprising the learnable embedding weights when encoded with the meta data.
3 . The image classification learning network according to claim 2 , wherein the embedding matrix is connected to the fully connected layer of neurons.
4 . The image classification learning network according to claim 1 , wherein a number of the learnable embedding weights are selectable by a user to influence accuracy of classification.
5 . The image classification learning network according to claim 1 , comprising a convolution layer configured to convolve the learned object provided by the embedding layer when concatenated with the image data.
6 . The image classification learning network according to claim 1 , comprising a convolution layer which follows a concatenation layer which combines the image data with the learned object.
7 . The image classification learning network according to claim 1 , comprising a convolution layer configured to provide convolved image data which are further concatenated with the learned object from the embedding layer.
8 . The image classification learning network according to claim 1 , comprising a convolution layer which leads a concatenation layer which combines the image data convolved by the convolution layer with the learned object.
9 . The image classification learning network according to claim 1 , where the standardized transmission or storage is configured to extract the image data and/or the meta data from a DICOM® object defined by an international standard for a communication, exchange, transmission and management of medical imaging information across various imaging modalities.
10 . The image classification learning network according to claim 1 , wherein the softmax layer is configured to provide a probability that a classification category characterizes an image or feature from the image data and meta data.
11 . An image classification learning method including:
inputting image data and meta data from a standardized transmission or storage;
transferring the meta data to an embedding layer wherein learnable embedding weights encode the meta data to provide a learned object; and
using a softmax layer to classify a combination of the image data and the learned object, wherein the embedding layer comprises a multiplication of a meta data vector comprising the meta data multiplying a weights matrix comprising the learnable embedding weights to create an embedding matrix, and wherein the weights matrix comprises one row of the learnable embedding weights to weight all elements of the meta data equally.