Secure cloud-based machine learning without sending original data to the cloud
Method and system for training a neural network. The neural network is split into first and second portions. A k-layer first portion is sent to a client training/inference engine and the second portion is retained by a server training/inference engine. At the splitting point, the kth layer is a one-way function in output computation has a number of nodes that are less than any other layer of the first portion. The client training/inference engine trains the first portion with input data in a set of training data. The server training/inference engine receives a batch of outputs from the client training and applies them to the second portion to train the entire neural network.
1. A method for training and using a neural network, the method comprising:
splitting the neural network into a first portion and a second portion, the first portion having k layers and the second portion having one or more subsequent layers of the neural network, based on selecting a kth layer of the first portion as a splitting point;
sending the first portion of the neural network to a client over a communications link, wherein, upon receipt of the first portion, the client applies one or more sets of inputs to the first portion of the neural network to create a batch of outputs;
receiving the batch of outputs from the client over the communications link; and
applying the batch of outputs received from the client to the second portion of the neural network to generate outputs.
2. The method of claim 1 ,
wherein the one or more sets of inputs are a plurality of training inputs available to the client; and
wherein the client applies the plurality of training inputs to the first portion of the neural network by forward-propagating each of the training inputs in the plurality of training inputs through the first portion of the neural network to generate the batch of outputs for the first portion of the neural network.
3. The method of claim 1 ,
wherein applying the batch of outputs received from the client to the second portion of the neural network includes training the second portion by:
forward-propagating each output in the batch of outputs received from the client through the second portion of the neural network to generate an output of the second portion; and
back propagating each output of the second portion through the layers of the second portion.
4. The method of claim 1 ,
wherein the one or more sets of inputs are one or more client inputs; and
wherein the client applies the one or more client inputs to the first portion of the neural network by performing an inference that includes forward-propagating each of the client inputs through the first portion of the neural network to generate the batch of outputs for the first portion of the neural network.
5. The method of claim 1 ,
wherein the one or more sets of inputs are one or more client inputs; and
wherein applying the batch of outputs received from the client to the second portion of the neural network includes performing an inference by forward-propagating each output in the batch of outputs received from the client through the second portion of the neural network to generate an output of the second portion.
6. The method of claim 1 ,
wherein receiving the batch of outputs from the client over the communications link includes receiving encrypted data representing the batch of outputs;
further comprising decrypting the encrypted data.
7. The method of claim 1 ,
wherein receiving the batch of outputs from the client over the communications link includes receiving encrypted data representing the batch of outputs and an encrypted seed along with the encrypted data;
further comprising:
sending the client a shared key table;
decrypting the encrypted seed and using the decrypted seed to access the shared key table to form a key for decrypting the encrypted data; and
decrypting the encrypted data with the key.
8. The method of claim 6 , wherein the encrypted data is encrypted using a key derived from a key table and a one-time pad.
9. The method of claim 6 , wherein the received encrypted data is secured using a secure socket layer protocol.
10. The method of claim 6 , wherein the received encrypted data is secured in a transaction of a block in a blockchain.
11. The method of claim 1 ,
wherein applying the batch of outputs received from the client to the second portion of the neural network includes training the neural network by:
forward-propagating each output in the batch of outputs received from the client through the second portion of the neural network to generate an output of the second portion; and
back propagating each output of the second portion through the layers of both the second and first portions without back propagating each output through the first portion sent to the client.
12. A server system comprising:
one or more CPUs;
a network interface coupled to the one or more CPUs and to a communications link; and
a RAM having loaded therein an operating system and one or more applications that are run by the one or more CPUs, wherein one of the one or more applications is configured to:
split a neural network into a first portion and a second portion, the first portion having k layers and the second portion having one or more subsequent layers of the neural network, based on selecting a kth layer of the first portion as a splitting point;
send the first portion of the neural network to a client over a communications link, wherein, upon receipt of the first portion, the client applies one or more sets of inputs to the first portion of the neural network to create a batch of outputs for the first portion of the neural network;
receive the batch of outputs from the client over the communications link; and
apply the batch of outputs received from the client to the second portion of the neural network to generate outputs for the second portion.
13. The system of claim 12 ,
wherein the one or more sets of inputs are a plurality of training inputs available to the client; and
wherein the client applies the plurality of training inputs to the first portion of the neural network by forward-propagating each of the training inputs in the plurality of training inputs through the first portion of the neural network to generate the batch of outputs.
14. The system of claim 12 ,
wherein applying the batch of outputs received from the client to the second portion of the neural network includes being configured to train the second portion by:
forward-propagating each output in the batch of outputs received from the client through the second portion of the neural network to create an output for the second portion; and
back propagating each output of the second portion through the layers of the second portion for a partial back propagation or back propagating each output of the second portion through the layers of both the second and first portion for a full back propagation.
15. The system of claim 12 ,
wherein the one or more sets of inputs are one or more client inputs; and
wherein the client applies the one or more client inputs to the first portion of the neural network by performing an inference that includes forward-propagating each of the client inputs through the first portion of the neural network to generate the batch of outputs.
16. The system of claim 12 ,
wherein the one or more sets of inputs are one or more client inputs; and
wherein applying the batch of outputs received from the client to the second portion of the neural network includes being configured to perform an inference by forward propagating each output in the batch of outputs received from the client through the second portion of the neural network to create an output for the second portion.
17. A non-transitory computer-readable medium containing instructions, which when executed by one or more processors, cause the processors to carry out a method for training and using a neural network, wherein the method comprises:
splitting the neural network into a first portion and a second portion, the first portion having k layers and the second portion having one or more subsequent layers of the neural network, based on selecting a kth layer of the first portion as a splitting point;
sending the first portion of the neural network to a client over a communications link, wherein, upon receipt of the first portion, the client applies one or more sets of inputs to the first portion of the neural network to create a batch of outputs for the first portion of the neural network;
receiving the batch of outputs from the client over the communications link; and
applying the batch of outputs received from the client to the second portion of the neural network to generate outputs for the second portion.
18. The computer-readable medium of claim 17 ,
wherein the one or more sets of inputs are a plurality of training inputs available to the client; and
wherein the client applies the plurality of training inputs to the first portion of the neural network by forward-propagating each of the training inputs in the plurality of training inputs through the first portion of the neural network to generate the batch of outputs.
19. The computer-readable medium of claim 17 ,
wherein applying the batch of outputs received from the client to the second portion of the neural network includes training the second portion by:
forward-propagating each output in the batch of outputs received from the client through the second portion of the neural network to generate an output of the second portion; and
back propagating each output of the second portion through the layers of the second portion for a partial back propagation or back propagating each output of the second portion through the layers of both the second and first portion for a full back propagation.
20. The computer-readable medium of claim 17 ,
wherein the one or more sets of inputs are one or more client inputs; and
wherein the client applies the one or more client inputs to the first portion of the neural network by performing an inference that includes forward-propagating each of the client inputs through the first portion of the neural network to generate the batch of outputs.
21. The computer-readable medium of claim 17 ,
wherein the one or more sets of inputs are a plurality of client inputs; and
wherein applying the batch of outputs received from the client to the second portion of the neural network includes performing an inference by forward-propagating each output in the batch of outputs received from the client through the second portion of the neural network to generate an output for the second portion.