Privacy-preserving machine learning

1. A system for performing privacy-preserving machine learning, the system comprising:

one or more processors; and

a computer readable medium storing a plurality of instructions that, when executed, control the one or more processors to perform a method comprising:

storing, at K training computers, secret-shared private data from a plurality of data clients, K being equal to two or more, wherein each data item of the secret-shared private data is represented by K parts when secret-shared, and wherein the secret-shared private data includes a set of training samples, each having d features and an output Y;

initializing values for d weights for a machine learning model, the d weights being secret-shared among the K training computers, wherein the d weights and the d features are stored as integers;

identifying a cost function that provides an accuracy of the d weights in predicting the outputs Y of the set of training samples;

selecting a training sample;

at each of the K training computers:

computing an error term of the cost function with respect to a Jth weight to determine a Jth delta part, wherein computing the error term includes:

performing a first multiplication using d feature parts of the selected training sample and d weight parts in determining a forward part, the forward parts comprising a secret-shared forward propagation result from a product of the d features and the d weights, wherein the first multiplication uses a first multiplication triplet of random integers;

truncating the Jth delta part;

using the truncated Jth delta part to update a Jth weight part of the d weight parts; and

repeating for other training samples and other weights of the d weights to train the machine learning model.

2. The system of claim 1 , wherein at least some of the d features are decimal values that are shifted to generate integers.

3. The system of claim 1 , wherein the machine learning model includes more weights than the d weights.

4. The system of claim 1 , wherein determining the Jth delta part further includes:

determining a difference part as part of the error term using the forward part and an output Y part, the difference parts comprising a secret-shared difference; and

performing a second multiplication of the difference part and a Jth feature part of a Jth feature to obtain a Jth delta part, the second multiplication using a second multiplication triplet of random integers.

5. The system of claim 4 , wherein the method further comprises:

truncating the difference part before the second multiplication.

6. The system of claim 4 , wherein the second multiplication includes multiplying the difference part by each of the d feature parts to obtain d delta parts of a delta part vector, the method further comprising:

truncating the d delta parts; and

using the truncated delta parts to update the d weight parts.

7. The system of claim 6 , wherein the training sample is selected as part of a batch of B training samples that is selected randomly, wherein the first multiplication uses a matrix X of B×d dimensions and an output Y vector of B outputs to form a forward part vector, and wherein the second multiplication uses the matrix X and a difference part vector of length d to form the delta part vector.

8. The system of claim 7 , wherein the method further comprises:

generating multiplication triplets as matrices using linearly homomorphic encryption (LHE) or oblivious transfer (OT), and wherein the first multiplication triplet and the second multiplication triplet share a matrix.

9. The system of claim 7 , wherein using the truncated Jth delta part to update the Jth weight part of the d weight parts includes using a learning step of α/|B|.

10. The system of claim 1 , wherein the method further comprises:

reconstructing the d weights using the d weight parts at each of the K training computers.

11. The system of claim 1 , wherein the method further comprises:

receiving a new sample having the d features;

applying, by each of the K training computers, weights parts of the machine learning model to obtain output parts; and

reconstructing, by the K training computers, the output parts to obtain a predicted output for the new sample.

12. A system for performing privacy-preserving machine learning, the system comprising:

one or more processors; and

a computer readable medium storing a plurality of instructions that, when executed, control the one or more processors to perform a method comprising:

storing, at K training computers, secret-shared private data from a plurality of data clients, K being equal to two or more, wherein each data item of the secret-shared private data is represented by K parts when secret-shared, and wherein the secret-shared private data includes a set of training samples, each having a set of d features X and an output Y;

initializing values for a set of d weights W for a machine learning model, the set of d weights W being secret-shared among the K training computers, wherein the set of d weights W and the set of d features X are decimal values that are shifted and stored as integers;

selecting a training sample I having d features X 1 ;

performing, using the K training computers, a first multiplication using X 1 and W to obtain a multiplication result u;

using a garbled circuit to evaluate, using the K training computers, a function F of the multiplication result u to obtain a forward propagation result Y 1 *, where the Function F is:

a first constant when u is less than a first cutoff,

a second constant when u is greater than a second cutoff, and

a monotonically changing function between the first constant and the second constant when u is between the first cutoff and the second cutoff;

determining, using the K training computers, an error term D using Y 1 * and an output Y 1 of the selected training sample;

performing, using the K training computers, a second multiplication of the error term D and a Jth element X I,J of X 1 to obtain a Jth delta Δ J ;

using the delta Δ J to update W J ; and

repeating for other training samples and other weights of the set of d weights W to train the machine learning model.

13. The system of claim 12 , wherein the training sample I is selected as part of a batch of B training samples that is selected randomly, wherein the first multiplication uses a matrix X of B×d dimensions and an output Y vector of B outputs, wherein the error term D is a vector of B values, and wherein the second multiplication uses the matrix X and the error term D to form a delta vector A for updating the set of d weights W.

14. The system of claim 12 , wherein using the garbled circuit to evaluate a function F of the forward propagation result Y 1 * includes:

determining whether Y 1 * is less than the first cutoff by:

receiving Y 1 * parts with one part added to the first constant;

adding the Y 1 * parts to obtain a first result; and

determining that Y 1 * is less than the first cutoff when the most significant bit of the first result is 0.

15. The system of claim 14 , wherein using the garbled circuit to evaluate a function F of the forward propagation result Y 1 * includes:

determining whether Y 1 * is less than the second cutoff by:

receiving Y 1 * parts with the first constant subtracted from one part;

adding the Y 1 * parts to obtain a second result; and

determining that Y 1 * is greater than the second cutoff when the most significant bit of the second result is 1.

16. The system of claim 15 , wherein the garbled circuit outputs values based on:

b 1 =the most significant bit of the first result, and

b 2 =the most significant bit of the second result, and wherein F includes (¬b 2 )+(b 2 Λ(¬b 1 )))Y 1 *.

17. The system of claim 16 , wherein the monotonically changing function is linear, where F include a constant C as a vertical intercept, and wherein F is provided as (¬b 2 )+(b 2 Λ(¬b 1 ))Y 1 *+C.

18. The system of claim 16 , wherein the garbled circuit outputs (¬b 2 ) and (b 2 Λ(¬b 1 )), and wherein F is evaluated using oblivious transfer.

19. A system for performing privacy-preserving machine learning, the system comprising:

one or more processors; and

a computer readable medium storing a plurality of instructions that, when executed, control the one or more processors to perform a method comprising:

storing, at K training computers, secret-shared private data from a plurality of data clients, K being equal to two or more, wherein each data item of the secret-shared private data is represented by K parts when secret-shared, and wherein the secret-shared private data includes a set of training samples, each having a set of d features X and an output Y;

initializing values for M sets of d weights W for a machine learning model, each set of d weights W corresponding to one of M nodes of a layer of the machine learning model, the set of d weights W being secret-shared among the K training computers, wherein the set of d weights W and the set of d features X are decimal values that are shifted and stored as integers;

selecting a training sample I having d features X 1 ;

performing, using the K training computers, a first multiplication using X 1 and W to obtain a multiplication result u 1 corresponding to a Lth node of M nodes of the machine learning model;

using a garbled circuit to evaluate, using the K training computers, a function F of multiplication result u 1 to obtain a forward propagation result Y I,L *, where the Function F is a ratio of:

a numerator function that is a first constant when u 1 exceeds a cutoff value C in one direction and is a monotonically changing function when u 1 exceeds the cutoff value C in the opposite direction; and

a denominator function that is a sum of all of the numerator functions of the M nodes;

determining, using the K training computers, an error term D using Y I,L * and an output Y 1 of the selected training sample;

performing, using the K training computers, a second multiplication of the error term D and a Jth element X I,J of X 1 to obtain a Jth delta Δ J ;

using the delta Δ J to update W J ; and

repeating for other training samples and other weights of the set of d weights W to train the machine learning model.

20. The system of claim 19 , wherein the M nodes are of a same layer of a neural network.