Method and system for securely storing data for use with artificial neural networks
View Patent ↗Systems and methods are disclosed for non-orthogonal encryption of data such that artificial neural networks are trained directly on the encrypted data and testing/inference is performed directly on encrypted data. For use with artificial neural networks, original input tensors are programmatically subjected to a sequence of encryption steps: a method for padding the feature set with randomly chosen sets of values; a method for perturbation of the data by altering elements of the feature set (the values stored in the dimensions of input vectors); a method that applies a fixed index shuffle of the data elements in the feature set or applies a random orthogonal matrix transformation of the data elements; and a method for applying convolutions (filters) and pooling (downsampling) to the fixed shuffled data such that informational structure is preserved.
1 . A method for using modified data with a neural network, the method comprising:
for each example of a plurality of examples, generating, by one or more processors executing stored instructions, a corresponding transformation of a plurality of transformations,
where each transformation of the plurality of transformations is a combination of one or more orthogonal transformations and one or more nonorthogonal transformations,
where the one or more orthogonal transformations are the same for each transformation of the plurality of transformations,
where each example is a training example of a plurality of training examples or a test example,
where each example of the plurality of examples includes a stored array of values having n elements,
where each transformation of the plurality of transformations includes a plurality of pads of values having p elements,
where at least one pad of values corresponding to at least one training example differs from each of the other pads of values corresponding to the other training examples,
where a modifying each example according to the corresponding transformation includes, by means of one or more processors executing stored instructions, an appending operation of appending the corresponding pad of values to the example and one or more orthogonal operations of applying the one or more orthogonal transformations to the example, to form a modified example having n+p elements, where each operation is applied to the values as presented to that operation,
such that each example that is modified undergoes a transformation that increases the size of the example from n elements to n+p elements, and
where the transformation includes at least one appending operation that precedes at least one of the one or more orthogonal operations such that the at least one orthogonal operation acts on an array of more than n elements, and such that at least one training example is modified with a different pad of values than the other training examples,
where the corresponding pad of values for each example comprises a plurality of non-zero values, and is generated from information that is not provided to the party performing the training of the neural network, and
wherein the one or more orthogonal transformations and the corresponding pad of values are each insufficient alone to recover the original example from the modified example, such that the original example is recoverable only given both the one or more orthogonal transformations and the corresponding pad of values, and wherein this mutual dependency is an intrinsic structural property of the transformation architecture itself, whereby the modified training examples may be used to train a neural network without providing the training party with sufficient information to reconstruct the original training examples, thereby improving the security of a neural network training system by enabling neural network training on proprietary or sensitive data without exposing the original data values to the party performing the training:
training the neural network by:
obtaining the plurality of training examples,
modifying each training example of the plurality of training examples according to the corresponding transformation to form a plurality of modified training examples, and
forming a trained neural network by training the neural network with the plurality of modified training examples; and
forming predictions using the trained neural network by:
accepting a test example, modifying the test example according to the corresponding transformation to form a modified test example, and
forming a prediction from the output of the trained neural network by providing the trained neural network with the modified test example as input, wherein training the neural network with the plurality of modified training examples produces a trained neural network that forms predictions on modified test examples without requiring the training party to access or reconstruct the original training examples or the pads of values at any point during training or prediction.
2 . The method of claim 1 , where each pad of values of the plurality of pads of values corresponding to a training example differs from each of the other pads of values corresponding to the other training examples.
3 . The method of claim 1 ,
where at least one nonorthogonal transformation of the one or more nonorthogonal transformations includes a plurality of perturbation functions, where each perturbation function of the plurality of perturbation functions corresponds to a position in the array of values as presented to the perturbation functions,
where the modifying each training example includes applying the corresponding plurality of perturbation functions to the values at the corresponding positions in the training example as presented to the perturbation functions, and
where the modifying the test example includes applying the corresponding plurality of perturbation functions to the values at the corresponding positions in the test example as presented to the perturbation functions.
4 . The method of claim 3 , where each perturbation function of the plurality of perturbation functions is the same.
5 . The method of claim 3 , where at least one perturbation function of the plurality of perturbation functions differs from each of the other perturbation functions of the plurality of perturbation functions, or where each perturbation function of the plurality of perturbation functions differs from each of the other perturbation functions of the plurality of perturbation functions.
6 . The method of claim 1 , where at least one orthogonal transformation of the one or more orthogonal transformations is an index shuffling or where at least one orthogonal transformation of the one or more orthogonal transformations is selected from a plurality of random orthogonal matrix transformations.
7 . The method of claim 1 , where training the neural network is performed by two or more parties.
8 . The method of claim 1 , where the forming predictions using the trained neural network is performed by two or more parties.
9 . The method of claim 1 , further including maintaining a ledger of the occurrence of modifying a test example or of the occurrence of providing the trained neural network with a modified test example to form a prediction.
10 . The method of claim 1 , further comprising:
prior to forming predictions using the trained neural network, encrypting the stored instructions for the plurality of transformations to form encrypted instructions, and
where the forming predictions using the trained neural network further includes accepting the encrypted instructions and decrypting the encrypted instructions to form instructions executable to modify the examples.
11 . A system for using modified data with a neural network, the system including networked memory and processors programmed to execute stored instructions to:
for each example of a plurality of examples, generate a corresponding transformation of a plurality of transformations,
where each transformation of the plurality of transformations is a combination of one or more orthogonal transformations and one or more nonorthogonal transformations,
where the one or more orthogonal transformations are the same for each transformation of the plurality of transformations,
where each example is a training example of a plurality of training examples or a test example,
where each example of the plurality of examples includes a stored array of values having n elements,
where each transformation of the plurality of transformations includes a plurality of pads of values having p elements,
where at least one pad of values corresponding to at least one training example differs from each of the other pads of values corresponding to the other training examples,
where a modifying each example according to the corresponding transformation includes, by means of one or more processors executing stored instructions, an appending operation of appending the corresponding pad of values to the example and one or more orthogonal operations of applying the one or more orthogonal transformations to the example, to form a modified example having n+p elements, where each operation is applied to the values as presented to that operation,
such that each example that is modified undergoes a transformation that increases the size of the example from n elements to n+p elements, and
where the transformation includes at least one appending operation that precedes at least one of the one or more orthogonal operations such that the at least one orthogonal operation acts on an array of more than n elements, and such that at least one training example is modified with a different pad of values than the other training examples,
where the corresponding pad of values for each example comprises a plurality of non-zero values, and is generated from information that is not provided to the party performing the training of the neural network, and
wherein the one or more orthogonal transformations and the corresponding pad of values are each insufficient alone to recover the original example from the modified example, such that the original example is recoverable only given both the one or more orthogonal transformations and the corresponding pad of values, and wherein this mutual dependency is an intrinsic structural property of the transformation architecture itself, whereby the modified training examples may be used to train a neural network without providing the training party with sufficient information to reconstruct the original training examples, thereby improving the security of a neural network training system by enabling neural network training on proprietary or sensitive data without exposing the original data values to the party performing the training:
the one or more processors further programmed to train the neural network by:
obtaining the plurality of training examples,
modifying each training example of the plurality of training examples according to the corresponding transformation to form a plurality of modified training examples, and
forming a trained neural network by training the neural network with the plurality of modified training examples; and
the one or more processors further programmed to form predictions using the trained neural network by:
accepting a test example,
modifying the test example according to the corresponding transformation to form a modified test example, and
forming a prediction from the output of the trained neural network by providing the trained neural network with the modified test example as input, wherein training the neural network with the plurality of modified training examples produces a trained neural network that forms predictions on modified test examples without requiring the training party to access or reconstruct the original training examples or the pads of values at any point during training or prediction.
12 . The system of claim 11 , where each pad of values of the plurality of pads of values corresponding to a training example differs from each of the other pads of values corresponding to other training examples.
13 . The system of claim 11 ,
where at least one nonorthogonal transformation of the one or more nonorthogonal transformations includes a plurality of perturbation functions, where each perturbation function of the plurality of perturbation functions corresponds to a position in the array of values as presented to the perturbation functions,
where the modifying each training example includes applying the corresponding plurality of perturbation functions to the values at the corresponding positions in the training example as presented to the perturbation functions, and
where the modifying the test example includes applying the corresponding plurality of perturbation functions to the values at the corresponding positions in the test example as presented to the perturbation functions.
14 . The system of claim 13 , where each perturbation function of the plurality of perturbation functions is the same.
15 . The system of claim 13 , where at least one perturbation function of the plurality of perturbation functions differs from each of the other perturbation functions of the plurality of perturbation functions, or where each perturbation function of the plurality of perturbation functions differs from each of the other perturbation functions of the plurality of perturbation functions.
16 . The system of claim 11 , where at least one orthogonal transformation of the one or more orthogonal transformations is an index shuffling or where at least one orthogonal transformation of the one or more orthogonal transformations is selected from a plurality of random orthogonal matrix transformations.
17 . The system of claim 11 , where training the neural network is performed by two or more parties.
18 . The system of claim 11 , where the forming predictions using the trained neural network is performed by two or more parties.
19 . The system of claim 11 , further including the processors programmed to store, in the memory, a ledger of the occurrence of modifying a test example or the occurrence of providing the trained neural network with a modified test example to form a prediction.
20 . The system of claim 11 , further comprising:
prior to forming predictions using the trained neural network, encrypting the stored instructions for the plurality of transformations to form encrypted instructions, and
where the forming predictions using the trained neural network further includes accepting the encrypted instructions and decrypting the encrypted instructions to form instructions executable to modify the examples.
21 . The method of claim 1 , where each pad of values is randomly generated independently for each corresponding training example and test example.
22 . The method of claim 1 , where a statistical distribution of values in each pad of the plurality of pads is constrained to match the statistical distribution of values in the plurality of examples.
23 . The system of claim 11 , where each pad of values is randomly generated independently for each corresponding training example and test example.
24 . The system of claim 11 , where a statistical distribution of values in each pad of the plurality of pads is constrained to match the statistical distribution of values in the plurality of examples.
25 . The method of claim 1 , wherein the test example is one of a plurality of test examples, and where at least one test example of the plurality of test examples is modified with a different pad of values than the other test examples of the plurality of test examples.
26 . The system of claim 11 , wherein the test example is one of a plurality of test examples, and where at least one test example of the plurality of test examples is modified with a different pad of values than the other test examples of the plurality of test examples.
27 . The method of claim 1 , wherein the modifying each example includes two or more appending operations, each appending operation appending a corresponding pad of values having a number of elements independent of the number of elements in each other pad of values, such that the modified example has n+p elements where p is the total number of padding values contributed by all appending operations.
28 . The system of claim 11 , wherein the modifying each example includes two or more appending operations, each appending operation appending a corresponding pad of values having a number of elements independent of the number of elements in each other pad of values, such that the modified example has n+p elements where p is the total number of padding values contributed by all appending operations.