Identifying and mitigating disparate group impact in differential-privacy machine-learned models
A model evaluation system evaluates the extent to which privacy-aware training processes affect the direction of training gradients for groups. A modified differential-privacy (“DP”) training process provides per-sample gradient adjustments with parameters that may be adaptively modified for different data batches. Per-sample gradients are modified with respect to a reference bound and a clipping bound. A scaling factor may be determined for each per-sample gradient based on the higher of the reference bound or a magnitude of the per-sample gradient. Per-sample gradients may then be adjusted based on a ratio of the clipping bound to the scaling factor. A relative privacy cost between groups may be determined as excess training risk based on a difference in group gradient direction relative to an unadjusted batch gradient and the adjusted batch gradient according to the privacy-aware training.
1. A system for evaluating differential privacy for groups in training data, comprising:
one or more processors; and
a non-transitory computer-readable medium having instructions executable by the one or more processors for:
identifying a batch of training data samples, the batch of training data samples including at least one training data sample from a plurality of data groups;
determining a set of per-sample gradients for training a computer model by applying the computer model with a set of current model parameter values to the batch of training data samples;
determining an unadjusted batch gradient by combining the set of per-sample gradients;
determining an adjusted batch gradient by applying a differential-privacy algorithm to the set of per-sample gradients, the differential-privacy algorithm modifying per-sample gradients;
determining an excess risk for one or more data groups of the differential-privacy algorithm based on a direction error of data samples associated with the one or more groups, the direction error describing a change in direction between the adjusted batch gradient and the unadjusted batch gradient; and
training the model based on the excess risk of the data group.
2. The system of claim 1 , wherein the excess risk for a group is determined based on an orthogonal matrix describing a direction difference between the adjusted batch gradient and the unadjusted batch gradient.
3. The system of claim 2 , wherein the direction error is determined based on:
η
t
〈
g
D
a
,
𝔼
[
g
¯
B
g
B
(
g
B
-
M
B
g
B
)
]
〉
+
η
t
2
2
𝔼
[
g
¯
B
2
g
B
2
(
(
M
B
g
B
)
T
H
ℓ
a
(
M
B
g
B
)
-
g
B
T
H
ℓ
a
g
B
)
]
in which:
η t is a learning rate,
g D a is an unadjusted group gradient,
g B is the unadjusted batch gradient,
g B is the adjusted batch gradient,
M B is the orthogonal matrix,
H l a is a Hessian of a loss function l evaluated over data samples of the group a, and
is an expectation taken over the data samples of the group.
4. The system of claim 1 , wherein the excess risk is a disparate group-group excess risk.
5. The system of claim 1 , wherein the excess risk is determined for a first group relative to a second group is based on a difference between a first angle measured for an unadjusted group gradient and the unadjusted batch gradient and a second angle measured for the unadjusted group gradient and the adjusted batch gradient.
6. The system of claim 1 , wherein the excess risk is determined for a first group relative to the second group according to:
𝔼
[
g
¯
B
(
cos
θ
B
a
-
cos
θ
¯
B
a
)
]
>
g
D
b
g
D
a
𝔼
[
g
¯
B
(
cos
θ
B
b
-
cos
θ
¯
B
b
)
]
+
𝔼
[
g
¯
B
2
]
g
D
a
Where
:
θ
B
k
=
∠
(
g
D
k
,
g
B
)
and
θ
¯
B
k
=
∠
(
g
D
k
,
g
¯
B
)
g D a is an unadjusted group gradient for the first group,
g D b is an unadjusted group gradient for the second group,
g B is an unadjusted batch gradient,
g B is an adjusted batch gradient, and
is an expectation taken over the data samples of the respective groups and batch.
7. A method for evaluating differential privacy for groups in training data, the method comprising:
identifying a batch of training data samples, the batch of training data samples including at least one training data sample from a plurality of data groups;
determining a set of per-sample gradients for training a computer model by applying the computer model with a set of current model parameter values to the batch of training data samples;
determining an unadjusted batch gradient by combining the set of per-sample gradients;
determining an adjusted batch gradient by applying a differential-privacy algorithm to the set of per-sample gradients, the differential-privacy algorithm modifying per-sample gradients;
determining an excess risk for one or more data groups of the differential-privacy algorithm based on a direction error of data samples associated with the one or more groups, the direction error describing a change in direction between the adjusted batch gradient and the unadjusted batch gradient; and
training the model based on the excess risk of the data group.
8. The method of claim 7 , wherein the excess risk for a group is determined based on an orthogonal matrix describing a direction difference between the adjusted batch gradient and the unadjusted batch gradient.
9. The method of claim 8 , wherein the direction error is determined based on:
η
t
〈
g
D
a
,
𝔼
[
g
¯
B
g
B
(
g
B
-
M
B
g
B
)
]
〉
+
η
t
2
2
𝔼
[
g
¯
B
2
g
B
2
(
(
M
B
g
B
)
T
H
ℓ
a
(
M
B
g
B
)
-
g
B
T
H
ℓ
a
g
B
)
]
in which:
η t is a learning rate,
g D a is an unadjusted group gradient,
g B is the unadjusted batch gradient,
g B is the adjusted batch gradient,
M B is the orthogonal matrix,
H l a is a Hessian of a loss function l evaluated over data samples of the group a, and
is an expectation taken over the data samples of the group.
10. The method of claim 7 , wherein the excess risk is a disparate group-group excess risk.
11. The method of claim 7 , wherein the excess risk is determined for a first group relative to a second group is based on a difference between a first angle measured for an unadjusted group gradient and the unadjusted batch gradient and a second angle measured for the unadjusted group gradient and the adjusted batch gradient.
12. The method of claim 7 , wherein the excess risk is determined for a first group relative to the second group according to:
𝔼
[
g
¯
B
(
cos
θ
B
a
-
cos
θ
¯
B
a
)
]
>
g
D
b
g
D
a
𝔼
[
g
¯
B
(
cos
θ
B
b
-
cos
θ
¯
B
b
)
]
+
𝔼
[
g
¯
B
2
]
g
D
a
Where
:
θ
B
k
=
∠
(
g
D
k
,
g
B
)
and
θ
¯
B
k
=
∠
(
g
D
k
,
g
¯
B
)
g D a is an unadjusted group gradient for the first group,
g D b is an unadjusted group gradient the second group,
g B is an unadjusted batch gradient,
g B is an adjusted batch gradient, and
is an expectation taken over the data samples of the respective groups and batch.
13. A non-transitory computer-readable medium for training a computer model with differential privacy and reduced group-group privacy disparity, the non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
identify a batch of training data samples, the batch of training data samples including at least one training data sample from a plurality of data groups;
determine a set of per-sample gradients for training a computer model by applying the computer model with a set of current model parameter values to the batch of training data samples;
determine an unadjusted batch gradient by combining the set of per-sample gradients;
determine an adjusted batch gradient by applying a differential-privacy algorithm to the set of per-sample gradients, the differential-privacy algorithm modifying per-sample gradients;
determine an excess risk for one or more data groups of the differential-privacy algorithm based on a direction error of data samples associated with the one or more groups, the direction error describing a change in direction between the adjusted batch gradient and the unadjusted batch gradient; and
train the model based on the excess risk of the data group.
14. The non-transitory computer-readable medium of claim 13 , wherein the excess risk for a group is determined based on an orthogonal matrix describing a direction difference between the adjusted batch gradient and the unadjusted batch gradient.
15. The non-transitory computer-readable medium of claim 13 , wherein the direction error is determined based on:
η
t
〈
g
D
a
,
𝔼
[
g
¯
B
g
B
(
g
B
-
M
B
g
B
)
]
〉
+
η
t
2
2
𝔼
[
g
¯
B
2
g
B
2
(
(
M
B
g
B
)
T
H
ℓ
a
(
M
B
g
B
)
-
g
B
T
H
ℓ
a
g
B
)
]
in which:
η t is a learning rate,
g D a is an unadjusted group gradient,
g B is the unadjusted batch gradient,
g B is the adjusted batch gradient,
M B is the orthogonal matrix,
H l a is a Hessian of a loss function l evaluated over data samples of the group a, and
is an expectation taken over the data samples of the group.
16. The non-transitory computer-readable medium of claim 13 , wherein the excess risk is a disparate group-group excess risk.
17. The non-transitory computer-readable medium of claim 13 , wherein the excess risk is determined for a first group relative to a second group is based on a difference between a first angle measured for an unadjusted group gradient and the unadjusted batch gradient and a second angle measured for the unadjusted group gradient and the adjusted batch gradient.
18. The non-transitory computer-readable medium of claim 13 , wherein the excess risk is determined for a first group relative to the second group according to:
𝔼
[
g
¯
B
(
cos
θ
B
a
-
cos
θ
¯
B
a
)
]
>
g
D
b
g
D
a
𝔼
[
g
¯
B
(
cos
θ
B
b
-
cos
θ
¯
B
b
)
]
+
𝔼
[
g
¯
B
2
]
g
D
a
Where
:
θ
B
k
=
∠
(
g
D
k
,
g
B
)
and
θ
¯
B
k
=
∠
(
g
D
k
,
g
¯
B
)
g D a is an unadjusted group gradient for the first group,
g D b is an unadjusted group gradient for the second group,
g B is an unadjusted batch gradient,
g B is an adjusted batch gradient, and
is an expectation taken over the data samples of the respective groups and batch.