Differential privacy dataset generation using generative models
Apparatuses, systems, and techniques to train a generative model based at least in part on a private dataset. In at least one embodiment, the generative model is trained based at least in part on a differentially private Sinkhorn algorithm, for example, using backpropagation with gradient descent to determine a gradient of a set of parameters of the generative models and modifying the set of parameters based at least in part on the gradient.
1. A method comprising:
using a machine learning model to generate a first dataset;
calculating a set of changes to apply to the first dataset to cause a set of characteristics of the first dataset to more closely match a corresponding set of characteristics of a second dataset;
modifying the set of changes based at least in part on noise information to generate a set of modified changes, the noise information corresponding to one or more constraints applied to the machine learning model; and
using the modified set of changes to update the machine learning model.
2. The method of claim 1 , wherein the set of changes is calculated by computing a Sinkhorn loss based at least in part on the first dataset and the second dataset.
3. The method of claim 2 , wherein calculating the set of changes further comprises calculating a gradient applied to data of the first dataset based at least in part on the Sinkhorn loss and a cost matrix corresponding to the first dataset and the second dataset.
4. The method of claim 3 , wherein modifying the set of changes further comprises clipping the gradient to reduce a vector magnitude of the gradient.
5. The method of claim 3 , wherein using the modified set of changes to update the machine learning model further comprises applying a function to the gradient.
6. The method of claim 1 , wherein the noise information further comprises Gaussian noise.
7. The method of claim 1 , wherein using the machine learning model to generate the first dataset further comprises selecting a class of the first dataset.
8. The method of claim 1 , wherein the second dataset further comprises a sampling of a private dataset.
9. The method of claim 8 , wherein the private dataset is maintained by a data curator and protected by a privacy barrier.
10. The method of claim 8 , wherein the method further comprises:
causing the machine learning model to generate training data that emulates the private dataset; and
providing the training data to train one or more generative adversarial networks.
11. The method of claim 1 , wherein using the modified set of changes to update the machine learning model further comprises modifying a set of parameters of the machine learning model.
12. The method of claim 11 , wherein the one or more constraints comprises a privacy constraint, and wherein modifying the set of parameters of the machine learning model further comprises modifying a gradient by a value determined based at least in part on the privacy constraint.
13. The method of claim 1 , wherein the first dataset further comprises a plurality of images.
14. The method of claim 1 , wherein the first dataset further comprises one or more class labels corresponding to the first dataset.
15. The method of claim 1 , wherein the set of changes is calculated by at least calculating a semi-debiased Sinkhorn loss based at least in part on the first dataset and the second dataset.
16. A processor comprising one or more circuits to generate a machine learning model trained, at least in part, by:
determining, based at least in part on noise information and a first set of changes to apply to a first set of data generated by the machine learning model to make the first set of data more closely match a second set of data, a second set of changes to make to the machine learning model; and
updating the machine learning model based at least in part on the second set of changes, wherein the second set of data comprises data sampled from a private dataset.
17. The processor of claim 16 , wherein the first set of changes further comprises calculating a primal form optimal transport plan based at least in part on the first set of data and a sample of the second set of data.
18. The processor of claim 17 , wherein calculating the primal form optimal transport plan further comprises calculating a Sinkhorn loss.
19. The processor of claim 17 , wherein calculating the primal form optimal transport plan further comprises calculating a semi-debiased Sinkhorn loss.
20. The processor of claim 16 , wherein determining the second set of changes further comprises performing backpropagation with gradient descent to generate a gradient of a set of parameters of the machine learning model.
21. A non-transitory machine-readable medium having stored thereon a set of instructions, as a result of being performed by one or more processors, cause the one or more processors to at least:
determine a first set of changes to make a set of characteristics of a first dataset to more closely match a corresponding set of characteristics of a second dataset, the first dataset generated based at least in part on a model;
determine a second set of changes to make to the model based at least in part on the first set of changes; and
use the second set of changes to update the model.
22. The non-transitory machine-readable medium of claim 21 , wherein the set of instructions that cause the one or more processors to determine the first set of changes further comprises instructions, as a result of being performed by the one or more processors, cause the one or more processors to determine a primal form optimal transport plan by at least calculating a Sinkhorn loss based at least in part on the first dataset and the second dataset.
23. The non-transitory machine-readable medium of claim 21 , wherein the set of instructions further comprises instructions, as a result of being performed by the one or more processors, cause the one or more processors to calculate a gradient of the first set of changes.
24. The non-transitory machine-readable medium of claim 23 , wherein the set of instructions that cause the one or more processors to calculate the gradient of the first set of changes further comprises instructions, as a result of being performed by the one or more processors, cause the one or more processors to clip the gradient to reduce a maximum value included in the gradient.
25. The non-transitory machine-readable medium of claim 24 , wherein the set of instructions further comprises instructions, as a result of being
performed by the one or more processors, cause the one or more processors to perform at least one of:
adding noise to the gradient;
selecting a class of the first dataset prior to generation by the model;
calculating a gradient of the second set of changes; and
modifying the gradient by a value.
26. The non-transitory machine-readable medium of claim 21 , wherein the first dataset further comprises a set of images.
27. The non-transitory machine-readable medium of claim 26 , wherein the set of images further comprises images of a plurality of classes.
28. The non-transitory machine-readable medium of claim 21 , wherein the model further comprises a parametric model that generates training data for training one or more neural networks; and
wherein the second set of changes further comprise changes to a set of parameters of the parametric model determined based at least in part on a gradient of the set of parameters calculated based at least in part on a Sinkhorn loss between the first dataset and the second dataset, where the second dataset is sampled from a private dataset.