Extracting explanations from attention-based models
Providing an explanation for model outcome can include receiving input data, and passing the input data through an attention-based neural network, where the attention-based neural network learns attention weights associated with contextual embeddings corresponding to tokens of the input data and predicts an outcome corresponding to the input data. Based on an attention weight associated with a contextual embedding corresponding to a token of the input data, a signed relevance score can be determined to associate with the token for quantifying the token's relevance to the outcome. Based on the signed relevance score, an explanation of the token's contribution toward or against the outcome can be provided. The signed relevance score can be computed as a gradient of loss with respect to the attention weight.
1 . A system comprising:
a processor;
a memory device coupled with the processor;
the processor configured to at least:
receive input data;
pass the input data through an attention-based neural network, wherein the attention-based neural network learns attention weights associated with contextual embeddings corresponding to tokens of the input data and predicts an outcome corresponding to the input data, the attention-based neural network generating the contextual embeddings through stacks of self-attention layers, the contextual embeddings being context-aware intermediate representations of the tokens of the input data;
based on an attention weight associated with a contextual embedding corresponding to a token of the input data, determine a signed relevance score to associate with the token for quantifying the token's relevance to the outcome, the signed relevance score providing a directional relevance, the signed relevance score computed as a negative value of a gradient of a loss with respect to the attention weight associated with the contextual embedding, a negative gradient indicating that the contextual embedding contributes toward the outcome, and a positive gradient indicating that the contextual embedding contributes against the outcome; and
based on the signed relevance score determined based on the attention weight associated with the contextual embedding corresponding to the token of the input data, provide a directional explanation of the token's contribution toward or against the outcome by explaining that the token's relevance to the outcome as quantified based on the signed relevance value is toward the outcome responsive to the signed relevance score being a positive value, and explaining that the token's relevance to the outcome as quantified based on the signed relevance value is against the outcome responsive to the signed relevance score being a negative value.
2 . The system of claim 1 , wherein the processor is further configured to determine the signed relevance score by multiplying the gradient with the attention weight.
3 . The system of claim 1 , wherein the processor is further configured to allocate the signed relevance score to the token by summing a relevance of all neurons in the contextual embedding, with each single neuron relevance being computed by back-propagation of a layer-wise neuron relevance.
4 . The system of claim 1 , wherein the processor is further configured to provide a test for accuracy of the explanation, the test testing for resiliency property.
5 . The system of claim 1 , wherein the processor is further configured to provide a test for accuracy of the explanation, the test testing for consistency property.
6 . The system of claim 1 , wherein the processor is further configured to provide a test for accuracy of the explanation testing for resiliency property, the test selecting a subset of tokens in the input data to replace, and further based on explanation performance resulting from the test, using the subset of tokens in an adversarial attack experiment of the attention-based neural network.
7 . The system of claim 1 , wherein the input data includes image data and the attention-based neural network is trained to generate captions associated with the image data.
8 . The system of claim 1 , wherein the input data includes a sequence of text in first language, wherein the attention-based neural network is trained to generate a translation of the sequence of text in second language.
9 . A method comprising:
receiving input data;
passing the input data through an attention-based neural network, wherein the attention-based neural network learns attention weights associated with contextual embeddings corresponding to tokens of the input data and predicts an outcome corresponding to the input data;
based on an attention weight associated with a contextual embedding corresponding to a token of the input data, determining a signed relevance score to associate with the token for quantifying the token's relevance to the outcome, the signed relevance score providing a directional relevance, the signed relevance score computed as a negative value of a gradient of a loss with respect to the attention weight associated with the contextual embedding, a negative gradient indicating that the contextual embedding contributes toward the outcome, and a positive gradient indicating that the contextual embedding contributes against the outcome; and
based on the signed relevance score determined based on the attention weight associated with the contextual embedding corresponding to the token of the input data, providing a directional explanation of the token's contribution to the outcome by explaining that the token's relevance to the outcome as quantified based on the signed relevance value is toward the outcome responsive to the signed relevance score being a positive value, and explaining that the token's relevance to the outcome as quantified based on the signed relevance value is against the outcome responsive to the signed relevance score being a negative value.
10 . The method of claim 9 , further including determining the signed relevance score by multiplying the gradient with the attention weight.
11 . The method of claim 9 , further including allocating the signed relevance score to the token by summing a relevance of all neurons in the contextual embedding, with each single neuron relevance being computed by back-propagation of a layer-wise neuron relevance.
12 . The method of claim 9 , further including providing a test for accuracy of the explanation, the test testing for resiliency property.
13 . The method of claim 9 , further including providing a test for accuracy of the explanation, the test testing for consistency property.
14 . The method of claim 9 , further including providing a test for accuracy of the explanation testing for resiliency property, the test selecting a subset of tokens in the input data to replace, and further based on explanation performance resulting from the test, using the subset of tokens in an adversarial attack experiment of the attention-based neural network.
15 . The method of claim 9 , wherein the input data includes image data and the attention-based neural network is trained to generate captions associated with the image data.
16 . The method of claim 9 , wherein the input data includes a sequence of text in first language, wherein the attention-based neural network is trained to generate a translation of the sequence of text in second language.
17 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a device to cause the device to:
receive input data;
pass the input data through an attention-based neural network, wherein the attention-based neural network learns attention weights associated with contextual embeddings corresponding to tokens of the input data and predicts an outcome corresponding to the input data;
based on an attention weight associated with a contextual embedding corresponding to a token of the input data, determine a signed relevance score to associate with the token for quantifying the token's relevance to the outcome, the signed relevance score providing a directional relevance, the signed relevance score computed as a negative value of a gradient of a loss with respect to the attention weight associated with the contextual embedding, a negative gradient indicating that the contextual embedding contributes toward the outcome, and a positive gradient indicating that the contextual embedding contributes against the outcome; and
based on the signed relevance score determined based on the attention weight associated with the contextual embedding corresponding to the token of the input data, provide a directional explanation of the token's contribution toward or against the outcome by explaining that the token's relevance to the outcome as quantified based on the signed relevance value is toward the outcome responsive to the signed relevance score being a positive value, and explaining that the token's relevance to the outcome as quantified based on the signed relevance value is against the outcome responsive to the signed relevance score being a negative value.
18 . The computer program product of claim 17 , wherein the device is further caused to determine the signed relevance score by multiplying the gradient with the attention weight.
19 . The computer program product of claim 17 , wherein the device is further caused to allocate the signed relevance score to the token by summing a relevance of all neurons in the contextual embedding, with each single neuron relevance being computed by back-propagation of a layer-wise neuron relevance.