Multi-modal deep learning of structured and non-structured data
A method for multi-modal deep learning is provided. The method comprises receiving input data from a database, wherein the input data comprises different data types. Responsive to receiving the input data, a number of corresponding neural extractors are identified to which to route the input data according to data type. Each neural extractor is specialized in extracting latent representations from a specific data type. The neural extractors, via a number of machine learning models, extract latent representations from the input data. The latent representations are then directed to a number of corresponding neural predictors, wherein each input data type is directed to a modality-specific neural predictor. The neural predictors then generate a number of output predictions for each input data type.
1 . A computer-implemented method for multi-modal deep learning, the method comprising:
receiving, by one or more processors, coupled with memory, input data from a database, wherein the input data comprises different data types, and wherein at least a portion of the input data is displayed via a display device;
responsive to receiving the input data, routing, by the one or more processors, the input data to a plurality of neural extractors based on the different data types, wherein a first neural extractor of the plurality of neural extractors is trained to extract a first latent representation from a first data type of the different data types and a second neural extractor is trained to extract a second latent representation from a second data type of the different data types, wherein the first data type corresponds to structured data and the second data type corresponds to unstructured data;
extracting, by the one or more processors, via one or more machine learning models, the first latent representation from the structured data using the first neural extractor and the second latent representation from the unstructured data using the second neural extractor;
routing, by the one or more processors, (i) the first latent representation to a first modality-specific neural predictor, (ii) the second latent representation to a second modality-specific neural predictor, and (iii) a combination of the first latent representation and the second latent representation to at least one neural predictor trained on a combination of the first data type and the second data type;
generating, by the one or more processors, using the first modality-specific neural predictor, the second modality-specific neural predictor, and the at least one neural predictor trained on the combination of the first data type and the second data type, a plurality of output predictions corresponding to the first data type, the second data type, and the combination of the first data type and the second data type; and
transmitting, by the one or more processors, to a client device, data to cause the client device to display the plurality of output predictions.
2 . The method of claim 1 , wherein a modality router identifies the plurality of neural extractors to which to route the input data according to the different data types.
3 . The method of claim 1 , wherein structured input are fed into machine learning models comprising a fully connected neural network and a recurrent neural network.
4 . The method of claim 1 , wherein non-structured input are fed into machine learning models comprising a convolutional neural network and a transformer-based machine learning model.
5 . The method of claim 1 , wherein the input data comprise at least one of:
human resources profiles;
service calls; or
historical payroll transactions.
6 . The method of claim 1 , wherein the plurality of output predictions comprises at least one of an:
earning code classification;
job level prediction;
business function classification; or
customer demand prediction.
7 . A system for multi-modal deep learning, the system comprising:
a storage device configured to store program instructions; and
one or more processors operably connected to the storage device and configured to execute the program instructions to cause the system to:
receive input data from a database, wherein the input data comprises different data types, and wherein at least a portion of the input data is displayed via a display device;
responsive to receiving the input data, route the input data to a plurality of neural extractors based on the different data types, wherein a first neural extractor of the plurality of neural extractors is trained to extract a first latent representation from a first data type of the different data types and a second neural extractor is trained to extract a second latent representation from a second data type of the different data types, wherein the first data type corresponds to structured data and the second data type corresponds to unstructured data;
extract, neural extractors via one or more machine learning models, the first latent representation from the structured data using the first neural extractor and the second latent representation from the unstructured data using the second neural extractor;
route (i) the first latent representation to a first modality-specific neural predictor, (ii) the second latent representation to a second modality-specific neural predictor, and (ii ii) a combination of the first latent representation and the second latent representation to at least one neural predictor trained on a combination of the first data type and the second data type;
generate, using the first modality-specific neural predictor, the second modality-specific neural predictor, and the at least one neural predictor trained on the combination of the first data type and the second data type, a plurality of output predictions corresponding to the first data type, the second data type, and the combination of the first data type and the second data type; and
transmit, to a client device, data to cause the client device to display the plurality of output predictions.
8 . The system of claim 7 , wherein a modality router identifies the plurality of neural extractors to which to route the input data according to the different data types.
9 . The system of claim 7 , wherein structured input are fed into machine learning models comprising a fully connected neural network and a recurrent neural network.
10 . The system of claim 7 , wherein non-structured input are fed into machine learning models comprising a convolutional neural network and a transformer-based machine learning model.
11 . The system of claim 7 , wherein the input data comprise at least one of:
human resources profiles;
service calls; or
historical payroll transactions.
12 . The system of claim 7 , wherein the plurality of output predictions comprises at least one of an:
earning code classification;
job level prediction;
business function classification; or
customer demand prediction.
13 . A computer program product for multi-modal deep learning, the computer program product comprising:
a computer-readable storage medium having program instructions embodied thereon to perform the steps of:
receiving input data from a database, wherein the input data comprises different data types, and wherein at least a portion of the input data is displayed via a display device;
responsive to receiving the input data, routing the input data to a plurality of neural extractors based on the different data types, wherein a first neural extractor of the plurality of neural extractors is trained to extract a first latent representation from a first data type of the different data types and a second neural extractor is trained to extract a second latent representation from a second data type of the different data types, wherein the first data type corresponds to structured data and the second data type corresponds to unstructured data;
extracting, via one or more machine learning models, the first latent representation from the structured data using the first neural extractor and the second latent representation from the unstructured data using the second neural extractor;
routing (i) the first latent representation to a first modality-specific neural predictor, (ii) the second latent representation to a second modality-specific neural predictor, and (iii) a combination of the first latent representation and the second latent representation to at least one neural predictor trained on a combination of the first data type and the second data type;
generating, using the first modality-specific neural predictor, the second modality-specific neural predictor, and the at least one neural predictor trained on the combination of the first data type and the second data type, a plurality of output predictions corresponding to the first data type, the second data type, and the combination of the first data type and the second data type; and
transmitting, to a client device, data to cause the client device to display the plurality of output predictions.
14 . The computer program product of claim 13 , wherein a modality router identifies the plurality of neural extractors to which to route the input data according to the different data types.
15 . The computer program product of claim 13 , wherein structured input are fed into machine learning models comprising a fully connected neural network and a recurrent neural network.
16 . The computer program product of claim 13 , wherein non-structured input are fed into machine learning models comprising a convolutional neural network and a transformer-based machine learning model.
17 . The computer program product of claim 13 , wherein the input data comprise at least one of:
human resources profiles;
service calls; or
historical payroll transactions.
18 . The computer program product of claim 13 , wherein the plurality of output predictions comprises at least one of an:
earning code classification;
job level prediction;
business function classification; or
customer demand prediction.