Systems, methods, and devices for melanoma pathology using one or more neural networks

1 . A system for melanoma pathology, comprising:

a memory configured to store instructions;

one or more processors;

a server device;

a plurality of data-specific neural networks (NNs), each NN comprising an input layer, one or more intermediate layers, and an output layer;

wherein the input layer comprises a plurality of input nodes for processing or encoding input data into a data shape receivable by the data-specific NN, each input node being configured to process the input data to produce a first weighted value provided to a first intermediate layer of the one or more intermediate layers;

wherein the system is configured to receive a plurality of prognosis inputs as the input data, the plurality of prognosis inputs comprising:

molecular data comprising genetic information, biological markers, and biological information from a patient biopsy sample; image data; patient data; and clinical notes data;

wherein each type of prognosis input of the plurality of prognosis inputs is applied to a data-specific NN of the plurality of data-specific NNs that is trained or modeled according to a corresponding type of information;

wherein outputs of the plurality of data-specific NNs are encoded according to a preselected encoding scheme and produced by the output layer;

and

wherein layers of the plurality of data-specific NNs are integrated such that a combined output of the plurality of data-specific NNs is produced by each NN output being concatenated or applied to the one or more intermediate layers to

produce a complete prognosis output from all of the prognosis inputs and the plurality of data-specific NNs.

2 . The system of claim 1 , wherein the image data is received by an image analysis NN of the plurality of data-specific NNs, and the preselected encoding scheme used by the imagine analysis NN comprises a feature vector comprising a data shape of the image information at the intermediate layer, and wherein the data shape includes a number of set of molecular data points being analyzed, and a number of features contained in each feature vector for each specific molecular data point.

3 . The system of claim 1 , wherein the patient biopsy sample comprises a patient skin sample.

4 . The system of claim 1 , wherein the complete prognosis output includes at least one of: an indication of whether the biopsy sample includes melanoma, whether the melanoma is benign or malignant, a recommendation for conducting a nearest node biopsy,

a likelihood of metastasis, or

a likelihood of recurrence of the melanoma.

5 . The system of claim 1 , wherein

the molecular data comprises a first data shape,

the image data comprises a second data shape that is different from the first data shape, and

the encoded molecular data concatenated to the image data at the intermediate layer of an image analysis NN comprises a third data shape that is different from the first data shape and the second data shape.

6 . The system of claim 5 , wherein the molecular data is received by a molecular NN of the plurality of data-specific NNs, and

the molecular NN comprises a one-dimensional NN, and

the image analysis NN comprises a multi-dimensional convolutional NN (CNN).

7 . The system of claim 5 , wherein the molecular data comprises a number of molecular features corresponding to a polymerase chain reaction (PCR) procedure of the patient biopsy sample.

8 . The system of claim 5 , wherein the molecular NN comprises a NN trained using molecular data from a plurality of patient biopsy samples.

9 . The system of claim 5 , wherein the encoded molecular data comprises a batch size and a number of molecular features of the patient biopsy sample.

10 . The system of claim 5 , wherein the image data comprises a batch size and a height (H), weight (W), and color (C) of the patient biopsy sample.

11 . The system of claim 5 , wherein each slide of a plurality of slides from the patient biopsy sample comprises a batch size and image features.

12 . The system of claim 5 , wherein the image analysis NN comprises a NN trained using image information of a plurality of patient biopsy samples.

13 . The system of claim 5 , wherein

a data shape of the encoded molecular information shares a same rank or dimensionality with a data shape of the image information at the intermediate layer of the image analysis NN.

14 . The system of claim 5 , wherein the intermediate layer of the image analysis NN comprises a pooling layer of a convolutional neural network (CNN).

15 . The system of claim 5 , wherein the image analysis NN comprises a random forest algorithm.

16 . The system of claim 1 , wherein the one or more processors cause the server device to:

receive patient data corresponding to the patient biopsy sample; apply the patient data to a patient information NN;

produce, based on the patient data and the patient information NN, patient output information;

generate encoded patient information by encoding the patient output information according to the preselected encoding scheme; and

concatenate the encoded patient information to the encoded molecular information and the image information at the intermediate layer of the image analysis NN.

17 . The system of claim 16 , wherein

the patient output information comprises an indication of a likelihood of metastasis of melanoma, and

concatenation of the encoded patient information to the image information comprises modifying, in accordance with the encoded patient information, a feature vector of the image analysis NN at the intermediate layer.

18 . The system of claim 16 , wherein

the molecular data comprises deoxyribonucleic acid (DNA) or ribonucleic acid (RNA) of the patient biopsy sample,

the image data comprises a plurality of whole slide images (WSI) of the patient biopsy sample, and

the patient data comprises electronic medical records of a patient corresponding to the patient biopsy sample.

19 . The system of claim 1 , wherein the one or more processors cause the server device to:

receive clinical notes data corresponding to the patient biopsy sample; apply the clinical notes data to a clinical note NN;

produce, based on the clinical notes data and the clinical note NN, clinical note output information;

generate encoded clinical notes data by encoding the clinical note output information according to the preselected encoding scheme; and

concatenate the encoded clinical notes data to the encoded molecular information and the image information at the intermediate layer of the image analysis NN.

20 . The system of claim 19 , wherein

the clinical note output information comprises an indication of a likelihood of metastasis of melanoma, and

concatenation of the encoded clinical note information to the image information comprises modifying, in accordance with the encoded clinical note information, a feature vector of the image analysis NN at the intermediate layer.

21 . The system of claim 19 , wherein

the molecular data comprises deoxyribonucleic acid (DNA) or ribonucleic acid (RNA) of the patient biopsy sample,

the image data comprises a plurality of whole slide images (WSI) of the patient biopsy sample, and

the clinical notes data comprises physician analysis information of the patient biopsy sample.

22 . A method for melanoma pathology, comprising:

receiving a plurality of prognosis inputs comprising: molecular data comprising genetic information, biological markers, and biological information from a patient biopsy sample; image data; patient data; and clinical notes data;

applying the plurality of prognosis inputs to a plurality of data-specific neural networks (NNs), wherein each type of prognosis input of the plurality of prognosis inputs is applied to a data-specific NN of the plurality of data-specific NNs that is trained or modeled according to a corresponding type of information;

processing the plurality of prognosis inputs as input data into an input layer of the plurality of data-specific NNs;

encoding outputs of the plurality of data-specific NNs according to a preselected encoding scheme;

concatenating the outputs of the plurality of data-specific NNs and applying the outputs to one or more intermediate layers of the plurality of data-specific NNs; and

producing a complete prognosis output from all of the prognosis inputs and the plurality of data-specific NNs.

23 . The method of claim 22 , wherein the image data is received by an image analysis NN of the plurality of data-specific NNs, and the preselected encoding scheme used by the imagine analysis NN comprises a feature vector comprising a data shape of the image information at the intermediate layer, and wherein the data shape includes a number of set of molecular data points being analyzed, and a number of features contained in each feature vector for each specific molecular data point.

24 . The method of claim 22 , wherein the patient biopsy sample comprises a patient skin sample and the prognosis output comprises an indication of a risk level of metastasis of melanoma.

25 . The method of claim 22 , wherein

the molecular output information comprises an indication of a risk level of metastasis of melanoma in the patient biopsy sample, and

concatenation of the encoded molecular data to the image data comprises modifying, in accordance with the encoded molecular information, a feature vector of the image analysis NN at the intermediate layer.

26 . The method of claim 22 , wherein

the molecular NN comprises a one-dimensional NN, and

the image analysis NN comprises a multi-dimensional convolutional NN (CNN).

27 . The method of claim 22 , further comprising:

receiving patient data corresponding to the patient biopsy sample; applying the patient data to a patient information NN;

producing, based on the patient data and the patient information NN, patient output information;

generating encoded patient data by encoding the patient output information according to the preselected encoding scheme; and

concatenating the encoded patient data to the encoded molecular data and the image data at the intermediate layer of the image analysis NN.

28 . The method of claim 27 , wherein

the patient output information comprises an indication of a risk level of metastasis of melanoma in the patient biopsy sample, and

concatenation of the encoded patient data to the image data comprises modifying, in accordance with the encoded patient information, a feature vector of the image analysis NN at the intermediate layer.

29 . The method of claim 22 , further comprising:

receiving clinical note information corresponding to the patient biopsy sample; applying the clinical notes data to a clinical note NN;

producing, based on the clinical note information and the clinical note NN, clinical note output information;

generating encoded clinical notes data by encoding the clinical note output information according to the preselected encoding scheme; and

concatenating the encoded clinical notes data to the encoded molecular data and the image data at the intermediate layer of the image analysis NN.

30 . The method of claim 29 , wherein

the clinical note output information comprises an indication of a risk level of metastasis of melanoma in the patient biopsy sample, and

concatenation of the encoded clinical notes data to the image data comprises modifying, in accordance with the encoded clinical note information, a feature vector of the image analysis NN at the intermediate layer.

31 . A non-transitory, computer-readable medium comprising:

one or more instructions that when performed by one or more processors, cause the one or more processors to:

receive a plurality of prognosis inputs comprising: molecular data comprising genetic information, biological markers, and biological information from a patient biopsy sample; image data; patient data; and clinical notes data;

apply the plurality of prognosis inputs to a plurality of data-specific neural networks (NNs), wherein each type of prognosis input of the plurality of prognosis inputs is applied to a data-specific NN of the plurality of data-specific NNs that is trained or modeled according to a corresponding type of information;

process the plurality of prognosis inputs as input data into an input layer of the plurality of data-specific NNs;

encode outputs of the plurality of data-specific NNs according to a preselected encoding scheme;

concatenate the outputs of the plurality of data-specific NNs and applying the outputs to one or more intermediate layers of the plurality of data-specific NNs; and

produce a complete prognosis output from all of the prognosis inputs and the plurality of data-specific NNs.

32 . The non-transitory, computer-readable medium of claim 31 , wherein the image data is received by an image analysis NN of the plurality of data-specific NNs, and the preselected encoding scheme used by the imagine analysis NN comprises a feature vector comprising a data shape of the image information at the intermediate layer, and wherein the data shape includes a number of set of molecular data points being analyzed, and a number of features contained in each feature vector for each specific molecular data point.

33 . The non-transitory, computer-readable medium of claim 31 , wherein the patient biopsy sample comprises a patient skin sample and the complete prognosis output comprises an indication of a risk level of metastasis of melanoma in the patient biopsy sample.

34 . The non-transitory, computer-readable medium of claim 31 , wherein the molecular output information comprises an indication of a likelihood of metastasis of melanoma in the patient biopsy sample, and

concatenation of the encoded molecular data to the image data comprises modifying, in accordance with the encoded molecular information, a feature vector of the image analysis NN at the intermediate layer.

35 . The non-transitory, computer-readable medium of claim 31 , wherein the molecular NN comprises a one-dimensional NN, and

the image analysis NN comprises a multi-dimensional convolutional NN (CNN).

36 . The non-transitory, computer-readable medium of claim 31 , wherein the one or more processors are further configured to:

receive patient data corresponding to the patient biopsy sample; apply the patient data to a patient information NN;

produce, based on the patient information and the patient information NN, patient output information;

generate encoded patient data by encoding the patient output information according to the preselected encoding scheme; and

concatenate the encoded patient data to the encoded molecular information and the image information at the intermediate layer of the image analysis NN.

37 . The non-transitory, computer-readable medium of claim 36 , wherein

the patient output information comprises an indication of a risk level of metastasis of melanoma in the patient biopsy sample, and

concatenation of the encoded patient data to the image data comprises modifying, in accordance with the encoded patient data, a feature vector of the image analysis NN at the intermediate layer.

38 . The non-transitory, computer-readable medium of claim 31 , wherein the one or more processors are further configured to:

receive clinical notes data corresponding to the patient biopsy sample; apply the clinical notes data to a clinical note NN;

produce, based on the clinical notes data and the clinical note NN, clinical note output information;

generate encoded clinical notes data by encoding the clinical note output information according to the preselected encoding scheme; and

concatenate the encoded clinical notes data to the encoded molecular data and the image data at the intermediate layer of the image analysis NN.

39 . The non-transitory, computer-readable medium of claim 38 , wherein the clinical note output information comprises an indication of a risk level of metastasis of melanoma in the patient biopsy sample, and

concatenation of the encoded notes data to the image data comprises modifying, in accordance with the encoded clinical note information, a feature vector of the image analysis NN at the intermediate layer.