Method and system for protecting and removing private information used in large language models
A method for generating adversarial data for use in a large language model (LLM) including receiving an input condition at a generator neural network, generating synthetic data responsive to the input condition, a distribution of the synthetic data being configured to diverge from a distribution of authentic data, receiving each of the input condition, the synthetic data, and the authentic data at a discriminator neural network, and classifying each of the synthetic data and the authentic data as being one of authentic or synthetic by the discriminator neural network.
1. A method for generating adversarial data for use in a large language model (LLM) comprising:
receiving an input condition at a generator neural network;
generating synthetic data responsive to the input condition by the generator neural network, a distribution of the synthetic data being configured to diverge from a distribution of authentic data;
receiving each of the input condition, the synthetic data, and the authentic data at a discriminator neural network; and
classifying each of the synthetic data and the authentic data as being one of authentic or synthetic by the discriminator neural network.
2. The method of claim 1 wherein:
the generator neural network comprises an adversarial objective function used in generating the synthetic data; and
the adversarial objective function is configured to at least one of increase the divergence between the distribution of the synthetic data and the distribution of the authentic data and maximize a statistical divergence between the distribution of the synthetic data and the distribution of the authentic data.
3. The method of claim 2 wherein the adversarial objective function comprises an adversarial loss term configured to at least one of increase the divergence between the distribution of the synthetic data and the distribution of the authentic data and maximize the divergence between the distribution of the synthetic data and the distribution of the authentic data.
4. The method of claim 3 wherein the adversarial objective function further comprises a utility function configured to weight one or more data fields comprised by at least one of the synthetic data and the authentic data.
5. The method of claim 4 wherein the adversarial loss term is defined by the equation:
Ladv
=
∑
[
(
1
-
U
(
x
)
)
*
Divergence
(
G
(
x
)
,
Real
(
x
)
]
where Ladv is the adversarial loss term, U(x) is the utility function, G(x) is the distribution of the synthetic data, Real(x) is the distribution of the authentic data, and Divergence (G(x), Real(x)) is the divergence between the distribution of the synthetic data and the distribution of the authentic data.
6. The method of claim 5 wherein the adversarial objective function further comprises a generator loss term that represents a generator loss introduced by the synthetic data when evaluated by the discriminator neural network, the generator loss term being defined by the adversarial loss term, a cross-entropy loss term of the discriminator neural network configured to quantify an effectiveness of the generator neural network to cause the discriminator neural network to misclassify synthetic data as authentic data, and a tunable loss parameter configured to balance the adversarial loss term with the cross-entropy loss term.
7. The method of claim 6 wherein the generator loss introduced by the synthetic data is defined by the equation:
LG
=
Ladv
+
λ
*
CrossEntropy
(
discriminator
)
where LG is the generator loss term, λ is the tunable loss parameter, and CrossEntropy(discriminator) is the cross-entropy loss term of the discriminator neural network.
8. The method of claim 7 wherein the generator loss term is further defined by at least one of a perplexity function of the generator neural network that determines an effectiveness of the generator neural network in matching a language distribution of the authentic data, a perplexity hyperparameter for controlling a relative contribution of the perplexity function to the generator loss, an extractability function of the generator neural network that determines a likelihood of the synthetic data being extracted from the LLM, and an extractability hyperparameter for controlling a relative contribution of the extractability function to the generator loss.
9. The method of claim 8 wherein the generator loss is defined by the equation:
LG
=
Ladv
+
λ
*
CrossEntropy
(
discriminator
)
+
β
*
Perplexity
(
G
(
x
)
)
+
γ
*
Extractability
(
G
(
x
)
)
where β is the perplexity hyperparameter, Perplexity (G(x)) is the perplexity function of the generator neural network, γ is the extractability hyperparameter, and Extractability(G(x)) is the extractability function of the generator neural network.
10. The method of claim 1 wherein the synthetic data and the authentic data comprises at least one of personally identifiable information, copyrighted content, financial information, or medical information.
11. The method of claim 1 wherein the generator neural network is trained on data extracted from the LLM.
12. The method of claim 1 wherein the generator neural network and the discriminator neural network are comprised by a conditional generative adversarial network.
13. A device for operating a conditional generative adversarial network for use in a large language model (LLM) comprising:
a non-transitory computer-readable storage medium having stored thereon:
a database of authentic data; and
a database of synthetic data; and
one or more processors configured to:
operate a generator neural network configured to:
receive an input condition; and
generate the synthetic data stored in the database of synthetic data responsive to the input condition, a distribution of the synthetic data being configured to diverge from a distribution of the authentic data; and
operate a discriminator neural network configured to:
receive each of the input condition, the synthetic data, and the authentic data at the discriminator neural network; and
classify each of the synthetic data and the authentic data as being one of authentic or synthetic by the discriminator neural network.
14. The device of claim 13 , wherein:
the generator neural network comprises an adversarial objective function used in generating the synthetic data; and
the adversarial objective function is configured to at least one of increase the divergence between the distribution of the synthetic data and the distribution of the authentic data and maximize a statistical divergence between the distribution of the synthetic data and the distribution of the authentic data.
15. The device of claim 14 , wherein the adversarial objective function comprises an adversarial loss term configured to at least one of increase the divergence between the distribution of the synthetic data and the distribution of the authentic data and maximize the divergence between the distribution of the synthetic data and the distribution of the authentic data.
16. The device of claim 15 , wherein the adversarial objective function further comprises a utility function configured to weight one or more data fields comprised by at least one of the synthetic data and the authentic data.
17. The device of claim 16 , wherein the adversarial loss term is defined by the equation:
Ladv
=
∑
[
(
1
-
U
(
x
)
)
*
Divergence
(
G
(
x
)
,
Real
(
x
)
]
where Ladv is the adversarial loss term, U(x) is the utility function, G(x) is the distribution of the synthetic data, Real(x) is the distribution of the authentic data, and Divergence(G(x), Real(x)) is the divergence between the distribution of the synthetic data and the distribution of the authentic data.
18. The device of claim 17 , wherein the adversarial objective function further comprises a generator loss term that represents a generator loss introduced by the synthetic data when evaluated by the discriminator neural network, the generator loss term being defined by the adversarial loss term, a cross-entropy loss term of the discriminator neural network configured to quantify an effectiveness of the generator neural network to cause the discriminator neural network to misclassify synthetic data as authentic data, and a tunable loss parameter configured to balance the adversarial loss term with the cross-entropy loss term.
19. The device of claim 18 , wherein the generator loss introduced by the synthetic data is defined by the equation:
LG
=
Ladv
+
λ
*
CrossEntropy
(
discriminator
)
where LG is the generator loss term, λ is the tunable loss parameter, and CrossEntropy (discriminator) is the cross-entropy loss term of the discriminator neural network.
20. The device of claim 19 , wherein the generator loss term is further defined by at least one of a perplexity function of the generator neural network that determines an effectiveness of the generator neural network in matching a language distribution of the authentic data, a perplexity hyperparameter for controlling a relative contribution of the perplexity function to the generator loss, an extractability function of the generator neural network for that determines a likelihood of the synthetic data being extracted from the LLM, and an extractability hyperparameter controlling a relative contribution of the extractability function to the generator loss.
21. The device of claim 20 wherein the generator loss is defined by the equation:
LG
=
Ladv
+
λ
*
CrossEntropy
(
discriminator
)
+
β
*
Perplexity
(
G
(
x
)
)
+
γ
*
Extractability
(
G
(
x
)
)
where β is the perplexity hyperparameter, Perplexity(G(x)) is the perplexity function of the generator neural network, γ is the extractability hyperparameter, and Extractability(G(x)) is the extractability function of the generator neural network.
22. The device of claim 13 , wherein the synthetic data and the authentic data comprises at least one of personally identifiable information, copyrighted content, financial information, or medical information.
23. The device of claim 13 , wherein the generator neural network is trained on data extracted from the LLM.
24. The device of claim 13 wherein the generator neural network and the discriminator neural network are comprised by a conditional generative adversarial network.