Method and device for denoising causal relation extraction based on multi-task collaborative learning
Disclosed is a causal relation extraction method, specifically relating to a method and a device for denoising causal relation extraction based on multi-task collaborative learning. The method includes: inputting a text to be processed into a task-sharing layer of a denoising causal relation extraction model based on multi-task collaborative learning for feature extraction to obtain an extraction result; inputting the extraction result into the task-specific layer of the denoising causal relation extraction model based on multi-task collaborative learning to perform part-of-speech tagging, chunking analysis, and causal extraction tasks, to obtain feature labels corresponding to the tasks; performing causal strength classification and causal relation extraction tasks; conducting causal strength representation learning based on counterfactual negative sample contrastive learning, using a gating generator to obtain channel gains and temperature control coefficients, which are applied to gated attention calculation, to realize causal strength classification and guide extraction of causal relations through causal strength.
1 . A method for denoising causal relation extraction based on multi-task collaborative learning, comprising:
inputting a text to be processed into a task-sharing layer of a denoising causal relation extraction model based on multi-task collaborative learning for feature extraction to obtain an extraction result;
inputting the extraction result into the task-specific layer of the denoising causal relation extraction model based on multi-task collaborative learning to perform a part-of-speech tagging task, a chunking analysis task, and a causal extraction task, and obtaining a feature label corresponding to the part-of-speech tagging task, a feature label corresponding to the chunking analysis task, and a feature label corresponding to the causal extraction task respectively, wherein a first layer of the denoising causal relation extraction model based on multi-task collaborative learning is obtained by parallel training of the part-of-speech tagging task, the chunking analysis task and the causal extraction task; and
applying a label attention mechanism to a causal strength classification task and a causal relation extraction task to obtain explicit feature representations of the two tasks; and conducting causal strength representation learning based on counterfactual negative sample contrastive learning, using a gating generator to obtain a channel gain and a temperature control coefficient, which are applied to gated attention calculation, so as to realize causal strength classification and guide the extraction of causal relations through causal strength.
2 . The method according to claim 1 , wherein the first layer of the denoising causal relation extraction model based on multi-task collaborative learning comprises:
a part-of-speech tagging task subnet, comprising a first task-sharing sublayer and a first task-specific sublayer, wherein the first task-sharing sublayer is used for feature extraction on the text to be processed to obtain a first extraction result, and the first task-specific sublayer is used for part-of-speech tagging based on the first extraction result to obtain a feature label corresponding to the part-of-speech tagging task;
a chunking analysis task subnet, comprising a second task-sharing sublayer and a second task-specific sublayer, wherein the second task-sharing sublayer is used for feature extraction on the text to be processed to obtain a second extraction result, and the second task-specific sublayer is used for chunking analysis based on the second extraction result to obtain a feature label corresponding to the chunking analysis task; and
a causal relation extraction task subnet, comprising a third task-sharing sublayer and a third task-specific sublayer, wherein the third task-sharing sublayer is used for feature extraction on the text to be processed to obtain a third extraction result, and the third task-specific sublayer is used for causal extraction based on the third extraction result to obtain a feature label corresponding to the causal extraction task,
wherein the task-sharing layer comprises the first task-sharing sublayer, the second task-sharing sublayer and the third task-sharing sublayer, and the task-specific layer comprises the first task-specific sublayer, the second task-specific sublayer and the third task-specific sublayer.
3 . The method according to claim 2 , wherein the first task-sharing sublayer comprises a first Bidirectional Encoder Representations from Transformers (BERT) semantic extraction layer and a first Bidirectional Long Short-Term Memory (BiLSTM) layer connected in sequence; the second task-sharing sublayer comprises a second BERT semantic extraction layer and a second BILSTM layer connected in sequence; and the third task-sharing sublayer comprises a third BERT semantic extraction layer and a third BiLSTM layer connected in sequence, wherein part of parameters of the first BERT semantic extraction layer, the second BERT semantic extraction layer and the third BERT semantic extraction layer are the same; and part of parameters of the first BILSTM layer, the second BILSTM layer and the third BiLSTM layer are the same.
4 . The method according to claim 2 , wherein the first task-specific sublayer comprises a first feature conversion layer and a first label prediction layer connected in sequence; the second task-specific sublayer comprises a second feature conversion layer and a second label prediction layer connected in sequence; and the third task-specific sublayer comprises a third feature conversion layer and a third label prediction layer connected in sequence.
5 . The method according to claim 3 , wherein the method further comprises:
performing iterative training on a first layer of a to-be-trained denoising causal relation extraction model based on multi-task collaborative learning to obtain the denoising causal relation extraction model based on multi-task collaborative learning,
wherein during the iterative training process, part of parameters of the part-of-speech tagging task subnet, the chunking analysis task subnet, and the causal relation extraction task subnet are iteratively pruned.
6 . The method according to claim 5 , wherein a loss function for iterative training of the first layer of the to-be-trained denoising causal relation extraction model based on multi-task collaborative learning is:
loss
task
=
{
loss
C
E
,
loss
pos
tagging
,
loss
chunk
analysis
}
;
loss
task
=
-
log
p
(
Y
true
❘
"\[RightBracketingBar]"
X
)
;
log
p
(
Y
true
❘
"\[RightBracketingBar]"
X
)
=
s
(
X
,
Y
true
)
-
log
∑
Y
~
∈
Y
x
e
s
(
X
,
Y
~
)
,
wherein Y true is used to represent a ground-truth label sequence, Y X is used to represent all possible label sequences, s(⋅) is used to represent a score of a label sequence, loss CE is used to represent a loss function generated by the causal extraction task, loss pos tagging is used to represent a loss function generated by the part-of-speech tagging task, and loss chunk analysis is used to represent a loss function generated by the chunking analysis task.
7 . The method according to claim 5 , wherein during a n th iterative training, a pruning rate for iteratively pruning part of parameters of the part-of-speech tagging task subnet, the chunking analysis task subnet, and the causal relation extraction task subnet satisfies:
p
=
(
1
-
α
)
1
n
,
wherein p is used to represent the pruning rate of each iteration, and α is used to represent a percentage of parameters retained in the part-of-speech tagging task subnet, the chunking analysis task subnet, and the causal relation extraction task subnet after the completion of iterative training.
8 . The method according to claim 1 , wherein a second layer of the denoising causal relation extraction model based on multi-task collaborative learning comprises a label attention module, a causal strength classification task, and a causal strength prior gated attention; the label attention module applies the label attention mechanism to the causal strength classification task and the causal relation extraction task to obtain explicit feature representations of the two tasks, comprising token representation H S for causal relation extraction and token representation H R for causal strength classification; causal relations are used to guide causal strength learning, a transformer-based attention mechanism is adopted to perform representation learning for causal strength classification, and causal strength representation H′ R that contains causal relations is obtained; the causal strength representation learning based on counterfactual negative sample contrastive learning is also adopted; and the causal strength prior gated attention embeds the learned causal strength representation into attention representation of causal relation extraction, so as to guide the extraction of causal relations.
9 . The method according to claim 1 , wherein matrix H S and matrix H R are respectively mapped to Query matrix Q S , Key matrix K S , Value matrix V S , as well as Query matrix Q R , Key matrix K R , Value matrix V R by using different linear projections; then, the representation C R of the causal strength classification task after attention calculation is computed; and the specific calculation of the causal strength representation H′ R that contains causal relations is shown in the following formula:
C
R
=
Softmax
(
Q
R
K
S
T
d
k
)
V
S
H
R
′
=
L
N
(
H
R
+
C
R
)
,
wherein LN represents a normalization function of this layer, and d k is a dimension of a key vector;
the causal strength representation learning based on counterfactual negative sample contrastive learning comprises: selecting a series of tokens with different meanings during training, then calculating representations through cosine similarity, and finding several representations with the smallest similarity as negative samples to implement contrastive learning, a contrastive loss being as follows:
ℒ
c
l
R
=
-
log
exp
(
ϕ
(
H
R
,
H
R
′
)
/
τ
)
∑
H
S
neg
∈
O
exp
(
ϕ
(
H
R
,
H
R
neg
)
/
τ
)
wherein O represents a feature set of negative samples, τ represents a temperature control coefficient, and φ represents a similarity function.
10 . The method according to claim 1 , wherein a calculation method for obtaining the channel gain and the temperature control coefficient by the gate generator of the causal strength prior gated attention is as follows:
γ
=
σ
(
W
g
H
R
+
b
g
)
β
=
soft
plus
(
W
β
H
R
+
b
β
)
wherein σ is a Sigmoid function, and softplus=log (1+e −x ), which is used to ensure that β is a continuous positive value; the channel gain γ acts as a feature-wise gating, and γ introduces conditional sparsity in the latent space through dimension-wise rescaling of Query and Key; a temperature coefficient β>0 acts as a distribution sharpness controller, and β enters the denominator of softmax in a multiplicative manner to directly adjust an attention entropy.
11 . The method according to claim 1 , wherein the gated attention calculation comprises: for the input query Q S , key K S , and value V S , sequentially performing dimension-wise gating, temperature scaling, and normalization to obtain a causal relation representation H′ S containing causal strength, with the calculation formulas as follows: Q′=γ⊙Q S , K′=γ⊙K S ; S=Q′K′/(√{square root over (dB)}); A=softmax(S), H′ S =AV S ; then, concatenating the two vectors H′ R ∈R n×d and H′ S ∈R n×d to obtain a new vector H SR ; then, for each word in the sentence, using a feature representation of the word to obtain semantic information of the words before and after the word; finally, implicitly fusing causal strength information and causal relation information through a feedforward neural network, and obtaining output vectors and of the two tasks respectively through a normalization function.
12 . The method according to claim 1 , wherein in the causal strength classification task, a max-pooling operation is applied to {tilde over (H)} R to obtain a semantic representation C of the sentence, which is used as an input to a Softmax function, and a label O R of the causal strength is obtained through the following formula, wherein w R and b R are respectively a weight matrix and bias term of the causal strength; and the calculation formula is as follows:
p
R
=
Softmax
(
w
R
C
+
b
R
)
O
R
=
arg
max
(
p
R
)
the causal classification task adopts the same crf decoder and corresponding loss function as those used in the first layer.
13 . A denoising causal relation extraction device based on multi-task collaborative learning, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor is configured to read the program in the memory to implement the steps in the method comprising:
a feature extraction module, comprising a Bidirectional Encoder Representations from Transformers (BERT) semantic extraction layer and a Bidirectional Long Short-Term Memory (BILSTM) layer, configured to input a text to be processed into a task-sharing layer of a denoising causal relation extraction model based on multi-task collaborative learning for feature extraction to obtain an extraction result; and a task-specific module, comprising a first layer and a second layer, wherein: the first layer is configured to input the extraction result into the task-specific layer of the denoising causal relation extraction model based on multi-task collaborative learning to perform a part-of-speech tagging task, a chunking analysis task, and a causal extraction task, and obtain a feature label corresponding to the part-of-speech tagging task, a feature label corresponding to the chunking analysis task, and a feature label corresponding to the causal extraction task respectively, wherein the denoising causal relation extraction model based on multi-task collaborative learning is obtained by parallel training of the part-of-speech tagging task, the chunking analysis task and the causal extraction task; and the second layer applies a label attention mechanism to a causal strength classification task and a causal relation extraction task to obtain explicit feature representations of the two tasks; and conducting causal strength representation learning based on counterfactual negative sample contrastive learning, using a gating generator to obtain a channel gain and a temperature control coefficient, which are applied to gated attention calculation, so as to realize causal strength classification and guide the extraction of causal relations through causal strength.
14 . An electronic device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor is configured to read the program in the memory to implement the steps in the method for denoising causal relation extraction based on multi-task collaborative learning according to claim 1 .
15 . A non-transitory readable storage medium for storing a program, wherein the program, when executed by a processor, implements the steps in the method for denoising causal relation extraction based on multi-task collaborative learning according to claim 1 .