Splitting neural networks on multiple edge devices to train on vertically distributed data
One example method includes a pipeline for a distributed neural network. The pipeline includes a first phase that identifies intersecting identifiers across datasets of multiple clients in a privacy preserving manner. The second phase includes a distributed neural network that includes a data receiving portion at each of the clients and an orchestrator portion at an orchestrator. The data receiving portions and the orchestrator portions communicate forward and backward passes to perform training without revealing the raw training data.
1. A method for training a distributed neural network, comprising:
identifying intersecting identifiers for datasets across clients, each of the clients including a data receiving portion of the distributed neural network, the datasets including training data for the distributed neural network, wherein the intersecting identifiers include identifers found in two or more datasets across clients;
receiving the training data from the clients as input into corresponding data receiving portions;
generating a forward pass at each of the data receiving portions;
receiving the forward passes at an orchestrator that includes an orchestrator portion of the distributed neural network;
calculating a loss at the orchestrator portion and propagating the loss through the distributed neural network using a backwards pass from the orchestrator portion to the data receiving portions; and
adjusting the neural network based on the backwards pass.
2. The method of claim 1 , wherein the forward pass, from each of the clients, includes an output from a last layer of the data receiving data portion, wherein the outputs are input into the first layer of the orchestrator portion of the distributed neural network.
3. The method of claim 1 , further comprising generating the loss from labels stored at the orchestrator and an output of a last layer of the orchestrator portion of the distributed neural network.
4. The method of claim 1 , further comprising identifying the intersecting identifiers associated with the clients with a privacy preserving mechanism such that none of non-intersecting identifiers are disclosed.
5. The method of claim 1 , further comprising sending a request, from the orchestrator to the clients, to load at least a batch of the training data into the data receiving portions of the neural network.
6. The method of claim 1 , further comprising the orchestrator providing the clients with a structure of the distributed neural network.
7. The method of claim 1 , further comprising training the distributed neural network with the training data.
8. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
identifying intersecting identifiers for datasets across clients, each of the clients including a data receiving portion of the distributed neural network, the datasets including training data for the distributed neural network, wherein the intersecting identifiers include identifers found in two or more datasets across clients;
receiving the training data from the clients as input into corresponding data receiving portions;
generating a forward pass at each of the data receiving portions;
receiving the forward passes at an orchestrator that includes an orchestrator portion of the distributed neural network;
calculating a loss at the orchestrator portion and propagating the loss through the distributed neural network using a backwards pass from the orchestrator portion to the data receiving portions; and
adjusting the neural network based on the backwards pass.
9. The non-transitory storage medium of claim 8 , wherein the forward pass, from each of the clients, includes an output from a last layer of the data receiving data portion, wherein the outputs are input into the first layer of the orchestrator portion of the distributed neural network.
10. The non-transitory storage medium of claim 8 , further comprising generating the loss from labels stored at the orchestrator and an output of a last layer of the orchestrator portion of the distributed neural network.
11. The non-transitory storage medium of claim 8 , further comprising identifying the intersecting identifiers associated with the clients with a privacy preserving mechanism such that none of non-intersecting identifiers are disclosed.
12. The non-transitory storage medium of claim 8 , further comprising sending a request, from the orchestrator to the clients, to load at least a batch of the training data into the data receiving portions of the neural network.
13. The non-transitory storage medium of claim 8 , further comprising the orchestrator providing the clients with a structure of the distributed neural network.
14. The non-transitory storage medium of claim 8 , further comprising training the distributed neural network with the training data.
15. A method for training a distributed neural network, comprising:
identifying intersecting identifiers for datasets across clients, each of the clients including a data receiving portion of the distributed neural network, the datasets including training data for the distributed neural network, wherein the intersecting identifiers include identifiers found in two or more datasets across clients;
receiving the training data from the clients as input into corresponding data receiving portions;
generating a forward pass at each of the data receiving portions;
receiving the forward passes at an orchestrator that includes an orchestrator portion of the distributed neural network;
sending an output of the orchestrator portion to the data receiving portions;
determining a loss at each of the data receiving portions;
sending the losses to the orchestrator portion;
propagating the loss through the distributed neural network using a backwards pass from the orchestrator portion to the data receiving portions; and
adjusting the neural network based on the backwards pass.
16. The method of claim 15 , wherein the forward pass, from each of the clients, includes an output from a last layer of the data receiving data portion, wherein the outputs are input into the first layer of the orchestrator portion of the distributed neural network, further comprising generating the loss from labels stored at the clients and an output of a last layer of the orchestrator portion.
17. The method of claim 15 , further comprising identifying the intersecting identifiers associated with the clients with a privacy preserving mechanism such that none of non-intersecting identifiers are disclosed.
18. The method of claim 15 , further comprising sending a request, from the orchestrator to the clients, to load at least a batch of the training data into the data receiving portions of the neural network and training the distributed neural network with the training data.
19. The method of claim 15 , further comprising the orchestrator providing the clients with a structure of the distributed neural network.
20. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform the method of claim 15 .