Neural network training from private data

1. A method of training a neural network model, the method comprising:

receiving, at a slave device, first configuration data for the neural network model from a master device, the master device being remote from the slave device, the master device including a first version of the neural network model;

instantiating, at the slave device, a second version of the neural network model using the first configuration data;

training, at the slave device, the second version of the neural network model using data from a first data source, the first data source being inaccessible by the master device;

receiving, at the master device, second configuration data for the neural network model, from the slave device, based on the trained second version of the neural network model;

instantiating, at the slave device, the second version of the neural network model as a student model;

instantiating, at the slave device, the first version of the neural network as a teacher model;

using, at the slave device, the teacher model to train the student model; and

generating the second configuration data to include parameters for the trained student model,

wherein the master device is configured to use the second configuration data to update parameters for the first version of the neural network model, and

wherein the first configuration data includes parameters for the first version of the neural network model and a neural network architecture definition for the second version of the neural network model.

2. The method of claim 1 further comprising:

loading, at the slave device, a binary executable; and

executing, using at least one processor of the slave device, the binary executable to implement at least the instantiating, training and outputting steps.

3. The method of claim 2 , wherein the binary executable includes the first configuration data.

4. A method of training a neural network model, the method comprising:

receiving, at a slave device, first configuration data for the neural network model from a master device, the master device being remote from the slave device, the master device including a first version of the neural network model;

instantiating, at the slave device, a second version of the neural network model using the first configuration data;

training, at the slave device, the second version of the neural network model using data from a first data source, the first data source being inaccessible by the master device;

receiving, at the master device, second configuration data for the neural network model, from the slave device, based on the trained second version of the neural network model;

instantiating, at the slave device, the second version of the neural network model as a student model;

instantiating, at the slave device, the first version of the neural network as a teacher model;

using, at the slave device, the teacher model to train the student model; and

generating the second configuration data to include parameters for the trained student model,

wherein the master device is configured to use the second configuration data to update parameters for the first version of the neural network model, and

wherein the first configuration data includes parameters for the first version of the neural network model and further comprising:

instantiating the student model and the teacher model using the parameters for the first version of the neural network model, such that the second version of the neural network model includes a copy of the first version of the neural network model; and

fixing, during training, the parameters of the teacher model while the parameters of the student model are trainable.

5. A method of training a neural network model, the method comprising:

receiving, at a slave device, first configuration data for the neural network model from a master device, the master device being remote from the slave device, the master device including a first version of the neural network model;

instantiating, at the slave device, a second version of the neural network model using the first configuration data;

training, at the slave device, the second version of the neural network model using data from a first data source, the first data source being inaccessible by the master device; and

receiving, at the master device, second configuration data for the neural network model, from the slave device, based on the trained second version of the neural network model,

wherein the master device is configured to use the second configuration data to update parameters for the first version of the neural network model,

wherein the first configuration data includes parameters for the first version of the neural network model and further comprising:

instantiating, using the parameters for the first version of the neural network model, the second version of the neural network that includes a trainable copy of the first version of the neural network model; and

generating, during the training, gradient data that is included in the second configuration data, and

wherein the master device is configured to update parameters for the first version of the neural network model by scaling its parameters by amounts indicated by the gradient data.

6. A slave device comprising:

a storage device to store parameter data for a second version of a neural network model;

memory to store computer program code configured to implement a distributed training operation;

a network interface to communicate data with a master device, the master device being remote and including a first version of the neural network model;

a storage interface to communicate with a first data source, the first data source being inaccessible by the master device; and

at least one processor configured to execute the computer program code stored in memory to:

receive, via the network interface, first configuration data for the neural network model from the master device;

instantiate the second version of the neural network model using the first configuration data;

perform a training operation to train the second version of the neural network model using data from the first data source obtained via the storage interface;

update the parameter data in the storage device based on the training operation;

output, via the network interface, second configuration data for the neural network model based on data resulting from the training operation;

instantiate the second version of the neural network model as a student model;

instantiate the first version of the neural network as a teacher model;

use the teacher model to train the student model; and

generate the second configuration data to include parameters for the trained student model,

wherein the second configuration data is used to update parameter data for the first version of the neural network model, and

wherein the first configuration data includes parameters for the first version of the neural network model and a neural network architecture definition for the second version of the neural network model.

7. The slave device of claim 6 , wherein the at least one processor further executes the computer program code in memory to:

load, at the slave device, a binary executable; and

execute, at the slave device, the binary executable to implement at least the instantiate of the second version of the neural network model using the first configuration data, to perform the training operation, and output.

8. The slave device of claim 7 , wherein the binary executable includes the first configuration data.

9. A slave device comprising:

a storage device to store parameter data for a second version of a neural network model;

memory to store computer program code configured to implement a distributed training operation;

a network interface to communicate data with a master device, the master device being remote and including a first version of the neural network model;

a storage interface to communicate with a first data source, the first data source being inaccessible by the master device; and

at least one processor configured to execute the computer program code stored in memory to:

receive, via the network interface, first configuration data for the neural network model from the master device;

instantiate the second version of the neural network model using the first configuration data;

perform a training operation to train the second version of the neural network model using data from the first data source obtained via the storage interface;

update the parameter data in the storage device based on the training operation;

output, via the network interface, second configuration data for the neural network model based on data resulting from the training operation;

instantiate the second version of the neural network model as a student model;

instantiate the first version of the neural network as a teacher model;

use the teacher model to train the student model; and

generate the second configuration data to include parameters for the trained student model,

wherein the second configuration data is used to update parameter data for the first version of the neural network model, and

wherein the first configuration data includes parameters for the first version of the neural network model and the at least one processor further executes the computer program code in memory to:

instantiate the student model and the teacher model using the parameters for the first version of the neural network model, such that the second version of the neural network model includes a copy of the first version of the neural network model; and

fix, during training, the parameters of the teacher model while the parameters of the student model are trainable.

10. The slave device of claim 9 , wherein the at least one processor further executes the computer program code in memory to:

load, at the slave device, a binary executable; and

execute, at the slave device, the binary executable to implement at least the instantiate of the second version of the neural network model using the first configuration data, to perform the training operation, and output.

11. The slave device of claim 10 , wherein the binary executable includes the first configuration data.

12. A slave device comprising:

a storage device to store parameter data for a second version of a neural network model;

memory to store computer program code configured to implement a distributed training operation;

a network interface to communicate data with a master device, the master device being remote and including a first version of the neural network model;

a storage interface to communicate with a first data source, the first data source being inaccessible by the master device; and

at least one processor configured to execute the computer program code stored in memory to:

receive, via the network interface, first configuration data for the neural network model from the master device;

instantiate the second version of the neural network model using the first configuration data;

perform a training operation to train the second version of the neural network model using data from the first data source obtained via the storage interface;

update the parameter data in the storage device based on the training operation; and

output, via the network interface, second configuration data for the neural network model based on data resulting from the training operation,

wherein the second configuration data is used to update parameter data for the first version of the neural network model, and

wherein the first configuration data includes parameters for the first version of the neural network model and the at least one processor further executes the computer program code in memory to:

instantiate, using the parameters for the first version of the neural network model, the second version of the neural network that includes a trainable copy of the first version of the neural network model; and

generate, during the training, gradient data that is included in the second configuration data, wherein the master device is configured to update parameters for the first version of the neural network model by scaling its parameters by amounts indicated by the gradient data.

13. The slave device of claim 12 , wherein the at least one processor further executes the computer program code in memory to:

load, at the slave device, a binary executable; and

execute, at the slave device, the binary executable to implement at least the instantiate of the second version of the neural network model using the first configuration data, to perform the training operation, and output.

14. The slave device of claim 13 , wherein the binary executable includes the first configuration data.