Method, system, and computer program product for synthetic oversampling for boosting supervised anomaly detection
Methods, systems, and computer program products may formulate an iterative data mix up problem into a Markov decision process (MDP) with a tailored reward signal to guide a learning process. To solve the MDP, a deep deterministic actor-critic framework may be modified to adapt a discrete-continuous decision space for training a data augmentation policy.
1. A method, comprising:
obtaining, with at least one processor, a training dataset X train including a plurality of source samples including a plurality of labeled normal samples and a plurality of labeled anomaly samples;
executing, with the at least one processor, a training episode by:
(i) initializing a timestamp t;
(ii) receiving, from an actor network π of an actor critic framework including the actor network π and a critic network Q, an action vector a t for the timestamp t, wherein the actor network π is configured to generate the action vector a t based on a state s t , wherein the state s t is determined based on a current pair of source samples of the plurality of source samples, and wherein the action vector a t includes a size of a nearest neighborhood k, a composition ratio α, a number of oversampling n, and a termination probability ∈;
(iii) combining the current pair of source samples according to the composition ratio α and the number of oversampling n to generate a labeled synthetic sample x syn associated with a label y syn ;
(iv) training, using the labeled synthetic sample x syn and the label y syn , a machine learning classifier ϕ;
(v) obtaining, based on the size of a nearest neighborhood k, source samples in the k-nearest neighborhood of the labeled synthetic sample x syn ;
(vi) generating, with the machine learning classifier ϕ, for the source samples in the k-nearest neighborhood of the labeled synthetic sample x syn and a subset of the plurality of source samples of the training dataset X train in a validation dataset X val , a plurality of classifier outputs;
(vii) selecting, from the source samples in the k-nearest neighborhood of the labeled synthetic sample x syn , a next pair of source samples;
(viii) storing, in a memory buffer, the state s t , the action vector a t , a next state s t+1 , and a reward r t , wherein the next state s t+1 is determined based on the next pair of source samples, and wherein the reward r t is determined based on the plurality of classifier outputs;
(ix) determining whether the termination probability ∈ satisfies a termination threshold;
(x) in response to determining that the termination probability ∈ fails to satisfy the termination threshold, incrementing the timestamp t, for a number of training steps S:
training the critic network Q according to a critic loss function that depends on the state s t , the action vector a t , and the reward r t ; and
training the actor network π according to an actor loss function that depends on an output of the critic network, and
after training the actor network π and the critic network Q for the number of training steps S, returning to step (ii) with the next pair of source samples as the current pair of source samples;
(xi) in response to determining that the termination probability ∈ satisfies the termination threshold, determining whether the number of training episodes executed satisfies a threshold number of training episodes;
(xii) in response to determining that the number of training episodes executed fails to satisfy the threshold number of training episodes, return to step (i) to execute a next training episode; and
(xiii) in response to determining that the number of training episodes executed satisfies the threshold number of training episodes, provide the machine learning classifier ϕ, wherein the plurality of source samples is associated with a plurality of transactions in a transaction processing network, wherein the plurality of labeled normal samples is associated with a plurality of non-fraudulent transactions of the plurality of transactions, and wherein the plurality of labeled anomaly samples is associated with a plurality of fraudulent transactions of the plurality of transactions;
receiving, with the at least one processor, transaction data associated with a transaction currently being processed in the transaction processing network:
processing, with the at least one processor, using the trained machine learning classifier ϕ, the transaction data to classify the transaction as a fraudulent or non-fraudulent transaction; and
in response to classifying the transaction as a fraudulent transaction, denying, with the at least one processor, authorization of the transaction in the transaction processing network.
2. The method of claim 1 , wherein the current pair of source samples are combined according to the composition ratio α to generate the labeled synthetic sample x syn according to the following Equations:
x syn =α*x 0 +(1−α)* x 1
y
syn
=
{
y
0
,
α
≥
0.5
y
1
,
otherwise
where x 0 is a first sample of the current pair of samples, x i is a second sample of the current pair of samples, y syn is a hard label for the labeled synthetic sample x syn , y 0 is a first hard label value, and y 1 is a second hard label value.
3. The method of claim 1 , wherein the reward r t is determined according to the following Equations:
Δ
ℳ
(
ϕ
t
)
=
ℳ
(
ϕ
t
(
𝒳
val
)
,
y
val
)
-
∑
i
=
t
-
m
t
-
1
ℳ
(
ϕ
i
(
𝒳
val
)
,
y
val
)
m
-
1
𝒞
(
ϕ
t
❘
"\[LeftBracketingBar]"
s
t
,
a
t
)
=
1
k
∑
i
=
0
k
P
(
y
i
=
0
❘
"\[LeftBracketingBar]"
x
i
,
ϕ
t
)
P
(
y
i
=
1
❘
"\[LeftBracketingBar]"
x
i
,
ϕ
t
)
where M is an evaluation metric, ΔM(ϕ t ) measures a performance improvement of the trained classifier ϕ t , X val is the validation data set, y val is a label set for the training data set, where
∑
i
=
t
-
m
t
-
1
ℳ
(
ϕ
i
(
𝒳
val
)
,
y
val
)
m
-
1
is a baseline for the timestamp t, m is a hyperparameter to define a buffer size for forming the baseline, C(οt|st, at) evaluates a model confidence of the trained classifier ϕ t , P is a model exploration function, k is the size of the nearest neighborhood specified by the action vector a t , x i is a k-nearest neighborhood of the labeled synthetic sample x syn in timestamp t, and y 1 is a label for x 1 .
4. The method of claim 1 , wherein the actor loss function is defined according to the following Equation:
L
π
(
θ
1
)
=
-
1
N
∑
i
=
1
N
Q
(
s
i
,
π
(
s
i
)
❘
"\[LeftBracketingBar]"
θ
2
)
where N is a number of transitions, π(s i |θ 2 ) is a projected action for a state s i , and Q(s i , π(s i )|θ 2 ) is an output of the critic network for the projected action π(s i |θ 2 ) and state s i , and
wherein the critic loss function is defined according to the following Equation:
L Q (θ 2 )=[ Q ( s t ,a t )− b t ] 2
where b t =R(s t , a t )+γQ(s t+1 , π(s t+1 |θ 1 )|θ 2 ), π(s t+1 |θ 1 ) is an action specified by the actor network, and γ is a decade factor.
5. The method of claim 1 , further comprising:
before executing the training episode:
training, with the at least one processor, using the training dataset X train the machine learning classifier ϕ, and
pre-computing, with the at least one processor, each k-nearest neighborhood for each source sample of the plurality of source samples in the training dataset X train .
6. A system, comprising:
at least one processor programmed and/or configured to:
obtain a training dataset X train including a plurality of source samples including a plurality of labeled normal samples and a plurality of labeled anomaly samples;
execute a training episode by:
(i) initializing a timestamp t;
(ii) receiving, from an actor network π of an actor critic framework including the actor network π and a critic network Q, an action vector a t for the timestamp t, wherein the actor network π is configured to generate the action vector a t based on a state s t , wherein the state s t is determined based on a current pair of source samples of the plurality of source samples, and wherein the action vector a t includes a size of a nearest neighborhood k, a composition ratio α, a number of oversampling n, and a termination probability ∈;
(iii) combining the current pair of source samples according to the composition ratio α and the number of oversampling n to generate a labeled synthetic sample x syn associated with a label y syn ;
(iv) training, using the labeled synthetic sample x syn and the label y syn , a machine learning classifier ϕ;
(v) obtaining, based on the size of a nearest neighborhood k, source samples in the k-nearest neighborhood of the labeled synthetic sample x syn ;
(vi) generating, with the machine learning classifier ϕ, for the source samples in the k-nearest neighborhood of the labeled synthetic sample x syn and a subset of the plurality of source samples of the training dataset X train in a validation dataset X val , a plurality of classifier outputs;
(vii) selecting, from the source samples in the k-nearest neighborhood of the labeled synthetic sample x syn , a next pair of source samples;
(viii) storing, in a memory buffer, the state s t , the action vector at, a next state s t +1, and a reward r t , wherein the next state s t+1 is determined based on the next pair of source samples, and wherein the reward r t is determined based on the plurality of classifier outputs;
(ix) determining whether the termination probability ∈ satisfies a termination threshold;
(x) in response to determining that the termination probability ∈ fails to satisfy the termination threshold, incrementing the timestamp t, for a number of training steps S:
training the critic network Q according to a critic loss function that depends on the state s t , the action vector a t , and the reward r t ; and
training the actor network π according to an actor loss function that depends on an output of the critic network, and
after training the actor network π and the critic network Q for the number of training steps S, returning to step (ii) with the next pair of source samples as the current pair of source samples;
(xi) in response to determining that the termination probability E satisfies the termination threshold, determining whether the number of training episodes executed satisfies a threshold number of training episodes;
(xii) in response to determining that the number of training episodes executed fails to satisfy the threshold number of training episodes, return to step (i) to execute a next training episode; and
(xiii) in response to determining that the number of training episodes executed satisfies the threshold number of training episodes, provide the machine learning classifier ϕ, wherein the plurality of source samples is associated with a plurality of transactions in a transaction processing network, wherein the plurality of labeled normal samples is associated with a plurality of non-fraudulent transactions of the plurality of transactions, and wherein the plurality of labeled anomaly samples is associated with a plurality of fraudulent transactions of the plurality of transactions;
receive transaction data associated with a transaction currently being processed in the transaction processing network;
process, using the trained machine learning classifier ϕ, the transaction data to classify the transaction as a fraudulent or non-fraudulent transaction; and
in response to classifying the transaction as a fraudulent transaction, deny authorization of the transaction in the transaction processing network.
7. The system of claim 6 , wherein the current pair of source samples are combined according to the composition ratio α to generate the labeled synthetic sample x syn according to the following Equations:
x syn =α*x 0 +(1−α)* x 1
y
syn
=
{
y
0
,
α
≥
0.5
y
1
,
otherwise
where x 0 is a first sample of the current pair of samples, x i is a second sample of the current pair of samples, y syn is a hard label for the labeled synthetic sample x syn , y 0 is a first hard label value, and y 1 is a second hard label value.
8. The system of claim 6 , wherein the reward re is determined according to the following Equations:
Δ
ℳ
(
ϕ
t
)
=
ℳ
(
ϕ
t
(
𝒳
val
)
,
y
val
)
-
∑
i
=
t
-
m
t
-
1
ℳ
(
ϕ
i
(
𝒳
val
)
,
y
val
)
m
-
1
𝒞
(
ϕ
t
❘
"\[LeftBracketingBar]"
s
t
,
a
t
)
=
1
k
∑
i
=
0
k
P
(
y
i
=
0
❘
"\[LeftBracketingBar]"
x
i
,
ϕ
t
)
P
(
y
i
=
1
❘
"\[LeftBracketingBar]"
x
i
,
ϕ
t
)
where M is an evaluation metric, ΔM(ϕ t ) measures a performance improvement of the trained classifier ϕ t , X val is the validation data set, y val is a label set for the training data set, where
∑
i
=
t
-
m
t
-
1
ℳ
(
ϕ
i
(
𝒳
val
)
,
y
val
)
m
-
1
is a baseline for the timestamp t, m is a hyperparameter to define a buffer size for forming the baseline, C(ϕ t |st, at) evaluates a model confidence of the trained classifier ϕ t , P is a model exploration function, k is the size of the nearest neighborhood specified by the action vector a t , x i is a k-nearest neighborhood of the labeled synthetic sample x syn in timestamp t, and y 1 is a label for x 1 .
9. The system of claim 6 , wherein the actor loss function is defined according to the following Equation:
L
π
(
θ
1
)
=
-
1
N
∑
i
=
1
N
Q
(
s
i
,
π
(
s
i
)
❘
"\[LeftBracketingBar]"
θ
2
)
where N is a number of transitions, π(s i |θ 2 ) is a projected action for a state s i , and Q(s i , π(s i )|θ 2 ) is an output of the critic network for the projected action π(s i |θ 2 ) and state s i , and
wherein the critic loss function is defined according to the following Equation:
L Q (θ 2 )=[ Q ( s t ,a t )− b t ] 2
where b t =R(s t , a t )+γQ(s t+1 , π(s t+1 |θ 1 )|θ 2 ), π(s t+1 |θ 1 ) is an action specified by the actor network, and γ is a decade factor.
10. The system of claim 6 , wherein the at least one processor is further programmed and/or configured to:
before executing the training episode:
train, using the training dataset X train , the machine learning classifier ϕ; and
pre-compute each k-nearest neighborhood for each source sample of the plurality of source samples in the training dataset X train .
11. A computer program product including a non-transitory computer readable medium including program instructions which, when executed by at least one processor, cause the at least one processor to:
obtain a training dataset X train including a plurality of source samples including a plurality of labeled normal samples and a plurality of labeled anomaly samples; and
execute a training episode by:
(i) initializing a timestamp t;
(ii) receiving, from an actor network π of an actor critic framework including the actor network π and a critic network Q, an action vector a t for the timestamp t, wherein the actor network π is configured to generate the action vector a t based on a state s t , wherein the state s t is determined based on a current pair of source samples of the plurality of source samples, and wherein the action vector a t includes a size of a nearest neighborhood k, a composition ratio α, a number of oversampling n, and a termination probability ∈;
(iii) combining the current pair of source samples according to the composition ratio α and the number of oversampling n to generate a labeled synthetic sample x syn associated with a label y syn ;
(iv) training, using the labeled synthetic sample x syn and the label y syn , a machine learning classifier ϕ;
(v) obtaining, based on the size of a nearest neighborhood k, source samples in the k-nearest neighborhood of the labeled synthetic sample x syn ;
(vi) generating, with the machine learning classifier ϕ, for the source samples in the k-nearest neighborhood of the labeled synthetic sample x syn and a subset of the plurality of source samples of the training dataset X train in a validation dataset X val , a plurality of classifier outputs;
(vii) selecting, from the source samples in the k-nearest neighborhood of the labeled synthetic sample x syn , a next pair of source samples;
(viii) storing, in a memory buffer, the state s t , the action vector a t , a next state s t +1, and a reward r t , wherein the next state s t+1 is determined based on the next pair of source samples, and wherein the reward r t is determined based on the plurality of classifier outputs;
(ix) determining whether the termination probability ∈ satisfies a termination threshold;
(x) in response to determining that the termination probability ∈ fails to satisfy the termination threshold, incrementing the timestamp t, for a number of training steps S:
training the critic network Q according to a critic loss function that depends on the state s t , the action vector a t , and the reward r t ; and
training the actor network π according to an actor loss function that depends on an output of the critic network, and
after training the actor network π and the critic network Q for the number of training steps S, returning to step (ii) with the next pair of source samples as the current pair of source samples;
(xi) in response to determining that the termination probability ∈ satisfies the termination threshold, determining whether the number of training episodes executed satisfies a threshold number of training episodes;
(xii) in response to determining that the number of training episodes executed fails to satisfy the threshold number of training episodes, return to step (i) to execute a next training episode; and
(xiii) in response to determining that the number of training episodes executed satisfies the threshold number of training episodes, provide the machine learning classifier ϕ, wherein the plurality of source samples is associated with a plurality of transactions in a transaction processing network, wherein the plurality of labeled normal samples is associated with a plurality of non-fraudulent transactions of the plurality of transactions, and wherein the plurality of labeled anomaly samples is associated with a plurality of fraudulent transactions of the plurality of transactions;
receive transaction data associated with a transaction currently being processed in the transaction processing network;
process, using the trained machine learning classifier ϕ, the transaction data to classify the transaction as a fraudulent or non-fraudulent transaction; and
in response to classifying the transaction as a fraudulent transaction, deny authorization of the transaction in the transaction processing network.
12. The computer program product of claim 11 , wherein the current pair of source samples are combined according to the composition ratio α to generate the labeled synthetic sample x syn according to the following Equations:
x syn =α*x 0 +(1−α)* x 0
y
syn
=
{
y
0
,
α
≥
0.5
y
1
,
otherwise
where x 0 is a first sample of the current pair of samples, x i is a second sample of the current pair of samples, y syn is a hard label for the labeled synthetic sample x syn , y 0 is a first hard label value, and y 1 is a second hard label value.
13. The computer program product of claim 11 , wherein the reward re is determined according to the following Equations:
Δ
ℳ
(
ϕ
t
)
=
ℳ
(
ϕ
t
(
𝒳
val
)
,
y
val
)
-
∑
i
=
t
-
m
t
-
1
ℳ
(
ϕ
i
(
𝒳
val
)
,
y
val
)
m
-
1
𝒞
(
ϕ
t
❘
"\[LeftBracketingBar]"
s
t
,
a
t
)
=
1
k
∑
i
=
0
k
P
(
y
i
=
0
❘
"\[LeftBracketingBar]"
x
i
,
ϕ
t
)
P
(
y
i
=
1
❘
"\[LeftBracketingBar]"
x
i
,
ϕ
t
)
where M is an evaluation metric, ΔM(ϕ t ) measures a performance improvement of the trained classifier ϕ t , X val is the validation data set, y val is a label set for the training data set, where
∑
i
=
t
-
m
t
-
1
ℳ
(
ϕ
i
(
𝒳
val
)
,
y
val
)
m
-
1
is a baseline for the timestamp t, m is a hyperparameter to define a buffer size for forming the baseline, C(ϕ t |st, at) evaluates a model confidence of the trained classifier ϕ t , P is a model exploration function, k is the size of the nearest neighborhood specified by the action vector a t , x i is a k-nearest neighborhood of the labeled synthetic sample x syn in timestamp t, and y 1 is a label for x 1 .
14. The computer program product of claim 11 , wherein the actor loss function is defined according to the following Equation:
L
π
(
θ
1
)
=
-
1
N
∑
i
=
1
N
Q
(
s
i
,
π
(
s
i
)
❘
"\[LeftBracketingBar]"
θ
2
)
where N is a number of transitions, π(s i |θ 2 ) is a projected action for a state s i , and Q(s i , π(s i )|θ 2 ) is an output of the critic network for the projected action π(s i |θ 2 ) and state s i , and
wherein the critic loss function is defined according to the following Equation:
L Q (θ 2 )=[ Q ( s t ,a t )− b t ] 2
where b t =R(s t , a t )+γQ(s t+1 , π(s t+1 |θ 1 )|θ 2 ), π(s t+1 |θ 1 ) is an action specified by the actor network, and γ is a decade factor.
15. The computer program product of claim 11 , wherein the program instructions, when executed by at least one processor, further cause the at least one processor to:
before executing the training episode:
train, using the training dataset X train , the machine learning classifier ϕ; and
pre-compute, each k-nearest neighborhood for each source sample of the plurality of source samples in the training dataset X train .