System, method, and computer program product for determining a reason for a deep learning model output
A system, method, and product for determining a reason for a deep learning model output that obtain training data associated with training samples and first labels for the training samples; train a first model using the training samples and the first labels, training the first model generating predictions for the training samples; train a second model using the training samples and the predictions as second labels for the training samples; extract one or more weights of the trained second model; process, using the first model, input data including features associated with at least one sample, to generate output data, the output data including at least one prediction for the at least one sample; and apply the one or more extracted weights to the features to determine one or more contributions of one or more features of the features to the at least one prediction for the at least one sample.
1 . A computer-implemented method comprising:
obtaining, with at least one processor, training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
training, with the at least one processor, a first neural network by:
providing, as input to the first neural network, the plurality of training samples,
receiving, as output from the first neural network, a plurality of first predictions for the plurality of training samples, wherein the plurality of first predictions includes a plurality of first probabilities that a plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more first weights of the first neural network according to a first objective function that depends on the plurality of first labels and the plurality of first predictions to generate a first trained neural network;
training, with the at least one processor, a second neural network that has fewer hidden layers than the first neural network by:
providing as input to the second neural network, the plurality of training samples,
receiving, as output from the second neural network, a plurality of second predictions for the plurality of training samples, wherein the plurality of second predictions includes a plurality of second probabilities that the plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more second weights associated with hidden units of at least one hidden layer of the second neural network according to a second objective function that depends on the plurality of first predictions as a plurality of second labels for the plurality of training samples and the plurality of second predictions to generate a second trained neural network;
extracting, with the at least one processor, from the second trained neural network, the one or more modified second weights associated with the hidden units of the at least one hidden layer of the trained second neural network;
obtaining, with the at least one processor, transaction data including a plurality of features x i associated with a transaction currently being processed in a transaction processing network;
processing, with the at least one processor, using the trained first neural network, the transaction data including the plurality of features x i associated with the transaction to generate a prediction for the transaction;
applying, by the at least one processor, in real-time during processing of the at least one transaction in the transaction processing network, the one or more modified second weights extracted from the trained second neural network to the plurality of features x i to generate a contribution score for each feature x i of the plurality of features x i to the prediction for the transaction generated by the first neural network by calculating a score for each path of a plurality of paths of each feature x i through the trained second neural network based on the one or more modified second weights extracted from the trained second neural network and at least one activation function, and summing each score for each path of the plurality of paths to determine the contribution score for that feature x i , and
providing, with the at least one processor, to a user device, in real-time during processing of the at least one transaction in the transaction processing network, the prediction and the contribution score for each feature x i as an indication of a contribution of that feature x i to the prediction.
2 . The computer-implemented method of claim 1 , wherein the first neural network includes a deep learning model including a greater number of hidden layers than the second neural network.
3 . The computer-implemented method of claim 1 , further comprising:
storing, with the at least one processor, in a memory, the one or more modified second weights removed from the trained second neural network.
4 . The computer-implemented method of claim 1 , wherein processing, using the trained first neural network, the transaction data including the plurality of features x i associated with the transaction includes:
providing, as input to the trained first neural network, the plurality of features x i associated with the transaction,
receiving, as output from the trained first neural network, output data including the prediction for the transaction, wherein the prediction includes a probability that the transaction is a fraudulent transaction; and
authorizing or denying, in the transaction processing network, based on the prediction including the probability that the transaction is a fraudulent transaction, the transaction.
5 . The computer-implemented method of claim 4 , wherein the one or more modified second weights include: weights for hidden units h j in a hidden layer of the trained second neural network, weights for activation units a j in an activation layer of the trained second neural network, weights for output units y m in an output layer of the trained second neural network, a weight matrix u ij applied to a feature x i on a path to a hidden unit h j of the trained second neural network, and a weight matrix v jm applied to an output of an activation unit a j on a path to an output unit y m .
6 . The computer-implemented method of claim 5 , wherein the contribution score Contribution (x i →y m ) for each feature x i of the plurality of features x i to the prediction for the transaction received as output from the first neural network is determined according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (x i →y m ) is the contribution score of the feature x i to the output unit y m , wherein K is a number of the plurality of features x i , wherein His a number of the hidden units h j in the hidden layer removed from the trained second neural network, wherein M is a number of the output units y m of the output layer removed from the trained second neural network, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein u ij is the weight matrix applied to the feature x i on the path to the hidden unit h j removed from the trained second neural network, wherein a j is the activation unit of the activation layer between the hidden layer and the output layer removed from the trained second neural network, and wherein v jm is the weight matrix applied to the output of the activation unit a j on the path to the output unit y m removed from the trained second neural network.
7 . A computing system comprising:
one or more processors programmed and/or configured to:
obtain training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
train a first neural network by:
providing, as input to the first neural network, the plurality of training samples,
receiving, as output from the first neural network, a plurality of first predictions for the plurality of training samples, wherein the plurality of first predictions includes a plurality of first probabilities that a plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more first weights of the first neural network according to a first objective function that depends on the plurality of first labels and the plurality of first predictions to generate a first trained neural network;
train a second neural network that has fewer hidden layers than the first neural network by:
providing as input to the second neural network, the plurality of training samples,
receiving, as output from the second neural network, a plurality of second predictions for the plurality of training samples, wherein the plurality of second predictions includes a plurality of second probabilities that the plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more second weights associated with hidden units of at least one hidden layer of the second neural network according to a second objective function that depends on the plurality of first predictions as a plurality of second labels for the plurality of training samples and the plurality of second predictions to generate a second trained neural network;
extract, from the second trained neural network, the one or more modified second weights associated with the hidden units of the at least one hidden layer of the trained second neural network;
obtain transaction data including a plurality of features x i associated with a transaction currently being processed in a transaction processing network;
process, using the trained first neural network, the transaction data including the plurality of features x i associated with the transaction to generate a prediction for the transaction;
apply in real-time during processing of the at least one transaction in the transaction processing network, using the one or more modified second weights extracted from the trained second neural network to the plurality of features x i to generate a contribution score for each feature x i of the plurality of features x i to the at least one prediction for the transaction generated by the first neural network by calculating a score for each path of a plurality of paths of each feature x i through the trained second neural network based on the one or more modified second weights extracted from the trained second neural network and at least one activation function, and summing each score for each path of the plurality of paths to determine the contribution score for that feature x i , and
provide, with the at least one processor, to a user device, in real-time during processing of the at least one transaction in the transaction processing network, the prediction and the contribution score for each feature x i as an indication of a contribution of that feature x i to the prediction.
8 . The computing system of claim 7 , wherein the first neural network includes a deep learning model including a greater number of hidden layers than the second neural network.
9 . The computing system of claim 7 , wherein the one or more processors are further programmed and/or configured to:
store, in a memory, the one or more modified second weights removed from the trained second neural network.
10 . The computing system of claim 7 , wherein the one or more processors process, using the trained first neural network, the transaction data including the plurality of features x i associated with the transaction by:
providing, as input to the trained first neural network, the plurality of features x i associated with the transaction,
receiving, as output from the trained first neural network, output data including a prediction for the transaction, wherein the prediction includes a probability that the transaction is a fraudulent transaction; and
authorizing or denying, in the transaction processing network, based on the prediction including the probability that the transaction is a fraudulent transaction, the transaction.
11 . The computing system of claim 10 , wherein the one or more modified second weights include: weights for hidden units h j in a hidden layer of the trained second neural network, weights for activation units a j in an activation layer of the trained second neural network, weights for output units y m in an output layer of the trained second neural network, a weight matrix u ij applied to a feature x i on a path to a hidden unit h j of the trained second neural network, and a weight matrix Vim applied to an output of an activation unit a j on a path to an output unit y m .
12 . The computing system of claim 11 , wherein the contribution score for each feature x i of the plurality of features x i to the at least one prediction for the transaction received as output from the first neural network is determined according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (x i →y m ) is the contribution score of the feature x i to the output unit y m , wherein K is a number of the plurality of features x i , wherein His a number of the hidden units h j in the hidden layer removed from the trained second neural network, wherein M is a number of the output units y m of the output layer removed from the trained second neural network, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein u ij is the weight matrix applied to the feature x i on the path to the hidden unit h j removed from the trained second neural network, wherein a j is the activation unit of the activation layer between the hidden layer and the output layer removed from the trained second neural network, and wherein v jm is the weight matrix applied to the output of the activation unit a j on the path to the output unit y m removed from the trained second neural network.
13 . A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
obtain training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
train a first neural network by:
providing, as input to the first neural network, the plurality of training samples,
receiving, as output from the first neural network, a plurality of first predictions for the plurality of training samples, wherein the plurality of first predictions includes a plurality of first probabilities that a plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more first weights of the first neural network according to a first objective function that depends on the plurality of first labels and the plurality of first predictions to generate a first trained neural network;
train a second neural network that has fewer hidden layers than the first neural network by:
providing as input to the second neural network, the plurality of training samples,
receiving, as output from the second neural network, a plurality of second predictions for the plurality of training samples, wherein the plurality of second predictions includes a plurality of second probabilities that the plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more second weights associated with hidden units of at least one hidden layer of the second neural network according to a second objective function that depends on the plurality of first predictions as a plurality of second labels for the plurality of training samples and the plurality of second predictions to generate a second trained neural network,
extract, from the second trained neural network, the one or more modified second weights associated with the hidden units of the at least one hidden layer of the trained second neural network;
obtain transaction data including a plurality of features x i associated with a transaction currently being processed in a transaction processing network;
process, using the trained first neural network, the transaction data including the plurality of features x i associated with the transaction to generate a prediction for the transaction;
apply in real-time during processing of the at least one transaction in the transaction processing network, using the one or more modified second weights extracted from the trained second neural network to the plurality of features x i to generate a contribution score for each feature x i of the plurality of features x i to the at least one prediction for the transaction generated by the first neural network by calculating a score for each path of a plurality of paths of each feature x i through the trained second neural network based on the one or more modified second weights extracted from the trained second neural network and at least one activation function, and summing each score for each path of the plurality of paths to determine the contribution score for that feature x i , and
provide, with the at least one processor, to a user device, in real-time during processing of the at least one transaction in the transaction processing network, the prediction and the contribution score Contribution (x i →y m ) for each feature x i as an indication of a contribution of that feature x i to the prediction.
14 . The computer program product of claim 13 , wherein the first neural network includes a deep learning model including a greater number of hidden layers than the second neural network.
15 . The computer program product of claim 13 , wherein the program instructions, when executed by at least one processor, further cause the at least one processor to:
store, in a memory, the one or more modified second weights removed from the trained second neural network.
16 . The computer program product of claim 13 , wherein the program instructions, when executed by at least one processor, further cause the at least one processor to process, using the trained first neural network, the transaction data including the plurality of features x i associated with the transaction by:
providing, as input to the trained first neural network, the plurality of features x i associated with the transaction,
receiving, as output from the trained first neural network, output data including a prediction for the transaction, wherein the prediction includes a probability that the transaction is a fraudulent transaction; and
authorizing or denying, in the transaction processing network, based on the prediction including the probability that the transaction is a fraudulent transaction, the transaction.
17 . The computer program product of claim 16 , wherein the one or more modified second weights include: weights for hidden units h j in a hidden layer of the trained second neural network, weights for activation units a j in an activation layer of the trained second neural network, weights for output units y m in an output layer of the trained second neural network, a weight matrix u ij applied to a feature x i on a path to a hidden unit h j of the trained second neural network, and a weight matrix Vim applied to an output of an activation unit a j on a path to an output unit y m .
18 . The computer program product of claim 17 , wherein the contribution score Contribution (x i →y m ) for each feature x i of the plurality of features x i to the prediction for the transaction received as output from the first neural network is determined according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (x i →y m ) is the contribution score of the feature x i to the output unit y m , wherein K is a number of the plurality of features x i , wherein His a number of the hidden units h j in the hidden layer removed from the trained second neural network, wherein M is a number of the output units y m of the output layer removed from the trained second neural network, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein u ij is the weight matrix applied to the feature x i on the path to the hidden unit h j removed from the trained second neural network, wherein a j is the activation unit of the activation layer between the hidden layer and the output layer removed from the trained second neural network, and wherein v jm is the weight matrix applied to the output of the activation unit a j on the path to the output unit y m removed from the trained second neural network.