Systems and methods for blind multimodal learning
A system and method are disclosed for providing a private multi-modal artificial intelligence platform. The method includes splitting a neural network into a first client-side network, a second client-side network and a server-side network and sending the first client-side network to a first client. The first client-side network processes first data from the first client, the first data having a first type. The method includes sending the second client-side network to a second client. The second client-side network processes second data from the second client, the second data having a second type. The first type and the second type have a common association. Forward and back propagation occurs between the client side networks and disparate data types on the different client side networks and the server-side network to train the neural network.
1. A method comprising:
creating, at a server device and based on assembled data from n client devices, a neural network having n bottom portions and a top portion, wherein the assembled data comprises different types of data;
transmitting, from the server device, each respective bottom portion of the n bottom portions to a respective client device of n client devices;
during a training iteration for training the neural network:
accepting, at the server device, a respective output from each respective bottom portion of the neural network to yield a plurality of respective outputs;
joining the plurality of respective outputs at a fusion layer on the server device to generate fused respective outputs; and
passing, from the server device, respective subsets of a set of gradients generated in a fusion layer from the server device to a respective client device of the n client devices, wherein each of the n client devices calculates a local set of gradients which is used to update local parameters associated with respective local models on the respective client device to yield a respective trained bottom portion of the neural network; and
after training, generating a combined model based on the respective trained bottom portion of the neural network from each respective client device.
2. The method of claim 1 , wherein each of the n bottom portions of the neural network specializes in being trained in a different data modality.
3. The method of claim 1 , wherein the different types of data comprise two or more of an image, a text, a video or a table.
4. The method of claim 1 , wherein each respective bottom portion of the neural network is trained on a respective different data modality.
5. The method of claim 1 , further comprising:
transmitting the combined model to a model owner associated with at least one of the server device, a respective client device or a separate computing device.
6. The method of claim 1 , wherein each respective bottom portion of the neural network is configured to be trained on the respective client device based on a different type of dataset.
7. The method of claim 1 , wherein joining the plurality of respective outputs at the fusion layer on the server device to generate fused respective outputs further comprises joining the plurality of respective outputs into one layer which is then passed as a first layer to the top portion of the neural network.
8. The method of claim 1 , wherein passing the respective subsets of the set of gradients of the fusion layer from the server device to each client device of the n client devices to generate the respective trained bottom portion of the neural network on each respective client device further comprises a back propagation process wherein the set of gradients is generated from a back propagation at the top portion of the neural network.
9. The method of claim 1 , wherein each respective bottom portion of the neural network and the top portion of the neural network, when combined, comprise a global model.
10. The method of claim 9 , wherein creating the n bottom portions of the neural network and creating the top portion of the neural network comprising splitting the global model into two parts.
11. The method of claim 1 , wherein each respective bottom portion of the neural network trained on the respective client device is different based on respective data available for training at each respective client device.
12. The method of claim 1 , wherein a data transformation occurs on at least one respective client device of the n client devices to enable the data to be consumable by each respective bottom portion of the neural network on each respective client device.
13. The method of claim 12 , wherein the data transformation comprises one or more of data transformation processes such as a resizing process or a normalization process.
14. The method of claim 1 , wherein combining the respective trained bottom portion of the neural network from each respective client device into the combined model is further performed utilizing a respective decorrelation loss value received from each respective client device of the n client devices.
15. A system comprising:
a processor; and
a computer-readable storage device storing instructions which, when executed by the processor, cause the processor to be configured to:
create, based on assembled data from n client devices, a neural network having n bottom portions and a top portion, wherein the assembled data comprises data of different types of data;
transmit each respective bottom portion of the n bottom portions to a respective client device of n client devices;
as part of a training iteration for training the neural network:
accept a respective output from each respective bottom portion of the neural network to yield a plurality of respective outputs;
join the plurality of respective outputs at a fusion layer to generate fused respective outputs; and
pass respective subsets of a set of gradients generated in a fusion layer to a respective client device of the n client devices, wherein each of the n client devices calculates a local set of gradients which is used to update local parameters associated with respective local models on the respective client device to yield a respective trained bottom portion of the neural network; and
after training, generate a combined model based on the respective trained bottom portion of the neural network from each respective client device.
16. The system of claim 15 , wherein each of the n bottom portions of the neural network specializes in being trained in a different data modality.
17. The system of claim 16 , wherein the different data modality comprises one or more of an image, text, a video or a table.
18. The system of claim 15 , wherein each respective bottom portion of the neural network is trained on a respective different data modality.
19. The system of claim 15 , wherein the computer-readable storage device stores additional instructions which, when executed by the processor, cause the processor to be configured to:
transmit the combined model to a model owner associated with at least one of the system, a respective client device or a separate computing device.