Systems and methods for providing a modified loss function in federated-split learning
Disclosed is a method that includes training, at a client, a part of a deep learning network up to a split layer of the client. Based on an output of the split layer, the method includes completing, at a server, training of the deep learning network by forward propagating the output received at a split layer of the server to a last layer of the server. The server calculates a weighted loss function for the client at the last layer and stores the calculated loss function. After each respective client of a plurality of clients has a respective loss function stored, the server averages the plurality of respective weighted client loss functions and back propagates gradients based on the average loss value from the last layer of the server to the split layer of the server and transmits just the server split layer gradients to the respective clients.
1. A method comprising:
training, at a client system of a plurality of client systems, a part of a deep learning network up to a split layer of the client system;
based on an output of the split layer of the client system, completing, at a server system, training of the deep learning network by asynchronously forward propagating the output received at a split layer of the server system to a last layer of the server system;
calculating a weighted loss function for the client system at the last layer of the server system to yield a calculated loss function for the client system;
storing the calculated loss function for the client system in a queue;
after each respective client system of the plurality of client systems has a respective loss function stored in the queue to yield a plurality of respective weighted client loss functions, averaging, at the server system, the plurality of respective weighted client loss functions to yield an average loss value;
back propagating gradients based on the average loss value from the last layer of the server system to the split layer of the server system to yield server system split layer gradients;
transmitting just the server system split layer gradients to the plurality of client systems, wherein no weights are shared across different client systems of the plurality of client systems;
back propagating, at the client system and from the split layer of the client system to an input layer of the client system, the server system split layer gradients to complete a training epoch of the deep learning network to yield a first model of a first client system of the plurality of client systems and a second model of a second client system of the plurality of client systems;
receiving, at an averaging component, a portion of the first model and a portion of the second model;
selecting, at the server system, a generator number g and a prime number p for a protocol and sending g and p to each client system, wherein g is a public base prime number and p is a public prime modulus;
generating, at each client system i, a random number r i and computing a key k i using a formula k i =g k mod p, wherein mod is a modulus and sending, from each client system i, the key k i to the server system to yield received keys k i ;
sending all the received keys k i to each client system;
computing, at each client system i with another client system j, a key with all other clients using a formula k ij =k j ri , wherein i and j are indices of the corresponding client systems and r i is a random number generated by client system i;
creating n shares of data for each client system i using a ShareGeneration (d) function: [d] i1 , . . . , [d] in =ShareGeneration (d) and masking a client system j share using the key k ij ;
computing, for each client system i, a client j share using a formula ([d′] ij =[d] ij +k ij ) for all 1≤j≤n and j≠i;
sending, from the client system i, the client system j share to the server system where [d′] ij is an encrypted share of d shared between client systems i and j;
sending the shares to each corresponding client system;
unmasking, at each client system, the received share with the key k ij as follows: [d] ij =[d′] ij −k ij ;
adding, at each client system, all respective local shares and sending the respective local shares to the server system;
adding, at the server system, all received respective local shares from each client system to yield a result, and dividing the result by n to yield the average model; and
distributing the average model to both the first client system and the second client system.
2. The method of claim 1 , wherein the weighted loss function comprises a minimizing of a statistical distance between (1) a distribution of activations communicated by the client system to the server system from just the split layer of the client system and (2) a classification loss.
3. The method of claim 2 , wherein the classification loss comprises a categorical cross-entropy or a cross-entropy.
4. The method of claim 1 , wherein storing the calculated loss function for the client system in the queue further comprises storing respective calculated loss function for each respective client system of the plurality of client systems.
5. The method of claim 4 , wherein storing respective calculated loss function for each respective client system of the plurality of client systems is performed asynchronously on a first-come-first-stored manner.
6. The method of claim 1 , wherein transmitting just the server system split layer gradients to the plurality of client systems further comprises transmitting just the server system split layer gradients to each client system of the plurality of client systems.
7. The method of claim 1 , wherein generating the average model is performed using secure multi-party computation.
8. A system comprising:
one or more hardware processors; and
a non-transitory computer-readable storage device configured to store instructions which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
training, at a client system of a plurality of client systems, a part of a deep learning network up to a split layer of the client system;
based on an output of the split layer of the client system, completing, at a server system, training of the deep learning network by asynchronously forward propagate the output received at a split layer of the server system to a last layer of the server system;
calculating a weighted loss function for the client system at the last layer of the server system to yield a calculated loss function for the client system;
storing the calculated loss function for the client system in a queue;
after each respective client system of the plurality of client systems has a respective loss function stored in the queue to yield a plurality of respective weighted client loss functions, averaging, at the server system, the plurality of respective weighted client loss functions to yield an average loss value;
back propagating gradients based on the average loss value from the last layer of the server system to the split layer of the server system to yield server system split layer gradients;
transmitting just the server system split layer gradients to the plurality of client systems, wherein no weights are shared across different client systems of the plurality of client systems;
back propagating, at the client system and from the split layer of the client system to an input layer of the client system, the server system split layer gradients to complete a training epoch of the deep learning network to yield a first model of a first client system of the plurality of client systems and a second model of a second client system of the plurality of client systems;
receiving, at an averaging component, a portion of the first model and a portion of the second model;
selecting, at the server system, a generator number g and a prime number p for a protocol and s ending g and p to each client system, wherein g is a public base prime number and p is a public prime modulus;
generating, at each client system i, a random number rj and computing a key ki using a formula ki=g*mod p, wherein mod is a modulus and sending, from each client system 1 , the key kj to the server system to yield received keys ki;
sending all the received keys kj to each client system;
computing, at each client system i with another client system j, a key with all other clients using a formula ki=kj″, wherein i and j are indices of the corresponding client systems and rj is a random number generated by client system i;
creating n shares of data for each client system i using a ShareGeneration (d) function: [d]i, [d]in=ShareGeneration (d) and masking a client system j share using the key ki;
computing, for each client system i, a client j share using a formula ([d]ij=[d]i, +ki) for all 1<j<n and j #i:
sending, from the client system i, the client system j share to the server system where [d′ ij is an encrypted share of d shared between client systems i and j:
sending the shares to each corresponding client system;
unmasking, at each client system, the received share with the key kij as follows: [dis_[d′ Ji˜ki:
adding, at each client system, all respective local shares and sending the respective local shares to the server system;
adding, at the server system, all received respective local shares from each client system to yield a result, and dividing the result by n to yield the average model:
and distributing the average model to both the first client system and the second client system.
9. The system of claim 8 , wherein the weighted loss function comprises a minimizing of a statistical distance between (1) a distribution of activations communicated by the client system to the server system from just the split layer of the client system and (2) a classification loss.
10. The system of claim 9 , wherein the classification loss comprises a categorical cross-entropy or a cross-entropy.
11. The system of claim 8 , wherein storing the calculated loss function for the client system in the queue further comprises storing respective calculated loss function for each respective client system of the plurality of client systems.
12. The system of claim 11 , wherein storing respective calculated loss function for each respective client system of the plurality of client systems is performed asynchronously on a first-come-first-stored manner.
13. The system of claim 8 , wherein transmitting just the server system split layer gradients to the client system further comprises transmitting just the server system split layer gradients to each client system of the plurality of client systems.
14. The system of claim 8 , wherein generating the average model is performed using secure multi-party computation.