Privacy-preserving machine learning

1. A method of performing privacy-preserving machine learning, the method comprising:

storing, at K training computers, secret-shared private data from a plurality of data clients, K being equal to two or more, wherein each data item of the secret-shared private data is represented by K parts when secret-shared, and wherein the secret-shared private data includes a set of training samples, each having d features and an output Y;

initializing values for a set of d weights for a machine learning model, the weights being secret-shared among the K training computers, wherein the weights and the features are stored as integers;

identifying a cost function that provides an accuracy of the set of d weights in predicting the outputs Y of the set of training samples;

selecting a training sample;

at each of the K training computers:

computing an error term of the cost function with respect to a Jth weight to determine a Jth delta part, wherein computing the error term includes:

performing a first multiplication using d feature parts of the selected training sample and d weight parts in determining a forward part, the forward parts comprising a secret-shared forward propagation result from a product of the d features and the d weights, the first multiplication using a first multiplication triplet of random integers;

truncating the Jth delta part;

using the truncated Jth delta part to update a Jth weight part of the d weight parts;

repeating for other training samples and other weights of the d weights to train the machine learning model.

2. The method of claim 1 , wherein at least some of the d features are decimal values that are shifted to generate integers.

3. The method of claim 1 , further comprising:

receiving a new sample having d features;

applying, by each of the K training computers, weights parts of the machine learning model to obtain output parts; and

reconstructing, by the K training computers, the output parts to obtain a predicted output for the new sample.

4. The method of claim 1 , wherein the machine learning model includes more weights than the set of d weights.

5. The method of claim 1 , wherein determining the Jth delta part further includes:

determining a difference part as part of the error term using the forward part and an output Y part, the difference parts comprising a secret-shared difference; and

performing a second multiplication of the difference part and a Jth feature part of a Jth feature to obtain a Jth delta part, the second multiplication using a second multiplication triplet of random integers.

6. The method of claim 5 , further comprising:

truncating the difference part before the second multiplication.

7. The method of claim 5 , wherein the second multiplication includes multiplying the difference part by each of the d feature parts to obtain d delta parts of a delta part vector, the method further comprising:

truncating the d delta parts; and

using the truncated delta parts to update the d weight parts.

8. The method of claim 7 , wherein the training sample is selected as part of a batch of B training samples that is selected randomly, wherein the first multiplication uses a matrix X of B×d dimensions and an output Y vector of B outputs to form a forward part vector, and wherein the second multiplication uses the matrix X and a difference part vector of length d to form the delta part vector.

9. The method of claim 8 , further comprising:

generating multiplication triplets as matrices using linearly homomorphic encryption (LHE) or oblivious transfer (OT), and wherein the first multiplication triplet and the second multiplication triplet share a matrix.

10. The method of claim 8 , wherein using the truncated Jth delta part to update the Jth weight part of the d weight parts includes using a learning step of α/|B|.

11. The method of claim 1 , further comprising:

reconstructing the d weights using the d weight parts at each of the K training computers.

12. A method of performing privacy-preserving machine learning, the method comprising:

storing, at K training computers, secret-shared private data from a plurality of data clients, K being equal to two or more, wherein each data item of the secret-shared private data is represented by K parts when secret-shared, and wherein the secret-shared private data includes a set of training samples, each having d features X and an output Y;

initializing values for a set of d weights W for a machine learning model, the weights being secret-shared among the K training computers, wherein the weights and the features are decimal values that are shifted and stored as integers;

selecting a training sample I having d features X I ;

performing, using the K training computers, a first multiplication using X I and W to obtain a multiplication result u;

using a garbled circuit to evaluate, using the K training computers, a function F of the multiplication result u to obtain a forward propagation result Y I *, where the Function F is:

a first constant when u is less than a first cutoff,

a second constant when u is greater than a second cutoff, and

a monotonically changing function between the first constant and the second constant when u is between the first cutoff and the second cutoff;

determining, using the K training computers, an error term D using Y I * and an output Y I of the selected training sample;

performing, using the K training computers, a second multiplication of the error term D and a Jth element X I,J of X I to obtain a Jth delta Δ J ;

using the delta Δ J to update W J ; and

repeating for other training samples and other weights of the d weights to train the machine learning model.

13. The method of claim 12 , wherein the training sample I is selected as part of a batch of B training samples that is selected randomly, wherein the first multiplication uses a matrix X of B×d dimensions and an output Y vector of B outputs, wherein the error term D is a vector of B values, and wherein the second multiplication uses the matrix X and the error term D to form a delta vector Δ for updating the d weights.

14. The method of claim 12 , wherein using the garbled circuit to evaluate a function F of the forward propagation result Y I * includes:

determining whether Y I * is less than the first cutoff by:

receiving Y I * parts with one part added to the first constant;

adding the Y I * parts to obtain a first result;

determining that Y I * is less than the first cutoff when the most significant bit of the first result is 0.

15. The method of claim 14 , wherein using the garbled circuit to evaluate a function F of the forward propagation result Y I * includes:

determining whether Y I * is less than the second cutoff by:

receiving Y I * parts with the first constant subtracted from one part;

adding the Y I * parts to obtain a second result;

determining that Y I * is greater than the second cutoff when the most significant bit of the second result is 1.

16. The method of claim 15 , wherein the garbled circuit outputs values based on:

b 1 =the most significant bit of the first result, and

b 2 =the most significant bit of the second result, and wherein F includes (¬b 2 )+(b 2 Λ(¬b 1 ))Y I *.

17. The method of claim 16 , wherein the monotonically changing function is linear, where F include a constant C as a vertical intercept, and wherein F is provided as (¬b 2 )+(b 2 Λ(¬b 1 ))Y I *+C.

18. The method of claim 16 , wherein the garbled circuit outputs (¬b 2 ) and (b 2 Λ(¬b 1 )), and wherein F is evaluated using oblivious transfer.

19. A method of performing privacy-preserving machine learning, the method comprising:

storing, at K training computers, secret-shared private data from a plurality of data clients, K being equal to two or more, wherein each data item of the secret-shared private data is represented by K parts when secret-shared, and wherein the secret-shared private data includes a set of training samples, each having d features X and an output Y;

initializing values for M sets of d weights W for a machine learning model, each set of d weights W corresponding to one of M nodes of a layer of the machine learning model, the weights being secret-shared among the K training computers, wherein the weights and the features are decimal values that are shifted and stored as integers;

selecting a training sample I having d features X I ;

performing, using the K training computers, a first multiplication using X I and W to obtain a multiplication result u I corresponding to a Lth node of M nodes of the machine learning model;

using a garbled circuit to evaluate, using the K training computers, a function F of multiplication result u I to obtain a forward propagation result Y I,L *, where the Function F is a ratio of:

a numerator function that is a first constant when u I exceeds a cutoff value C in one direction and is a monotonically changing function when u I exceeds the cutoff value C in the opposite direction; and

a denominator function that is a sum of all of the numerator functions of the M nodes;

determining, using the K training computers, an error term D using Y I,L * and an output Y I of the selected training sample;

performing, using the K training computers, a second multiplication of the error term D and a Jth element X I,J of X I to obtain a Jth delta Δ J ;

using the delta Δ J to update W J ; and

repeating for other training samples and other weights of the d weights to train the machine learning model.

20. The method of claim 19 , wherein the M nodes are of a same layer of a neural network.