Coreference resolution method, and method and apparatus for training coreference resolution model
A coreference resolution method, and a method and apparatus for training a coreference resolution model are provided. The coreference resolution method includes: acquiring a current utterance to be processed; inputting the current utterance into a coreference resolution detection sub-model to obtain a predicted insertion position where there is a semantic absence of the current utterance and/or a predicted deletion position of a word to be replaced; and inputting the predicted insertion position and/or the predicted deletion position of the current utterance, and a historical conversation corresponding to the current utterance into a resolution completion sub-model of the coreference resolution model to obtain a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and/or a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position.
1 . A coreference resolution method, comprising:
acquiring a current utterance to be processed;
inputting the current utterance into a coreference resolution detection sub-model of a coreference resolution model to obtain at least one of a predicted insertion position where there is a semantic absence of the current utterance and a predicted deletion position of a word to be replaced; and
inputting at least one of the predicted insertion position and the predicted deletion position of the current utterance, and a historical conversation corresponding to the current utterance into a resolution completion sub-model of the coreference resolution model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position;
wherein the inputting at least one of the predicted insertion position and the predicted deletion position of the current utterance, and a historical conversation corresponding to the current utterance into a resolution completion sub-model of the coreference resolution model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position, comprises:
calculating at least one of a mean vector of a starting position and an ending position of the predicted insertion position and a mean vector of a starting position and an ending position of the predicted deletion position of the current utterance;
taking the mean vector as a condition, performing semantic processing on an encoded vector obtained by encoding the current utterance by a transformer encoder by means of conditional layer normalization to obtain a conditional layer normalized semantic output variable; and
inputting the semantic output variable and the historical utterance of the current utterance into the resolution completion sub-model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position.
2 . The method according to claim 1 , wherein the coreference resolution detection sub-model comprises two binary classification networks, wherein one binary classification network is input with the current utterance and outputs a 01 sequence for indicating whether the position of each word in the current utterance is a starting position or an ending position of the predicted insertion position, and the other binary classification network is input with the current utterance and outputs a 01 sequence for indicating whether the position of each word in the current utterance is a starting position or an ending position of the predicted deletion position.
3 . The method according to claim 2 , wherein the binary classification network uses the following formula to determine whether the position of each word in the current utterance is a starting position or an ending position of the predicted insertion position or the predicted deletion position:
p
l
s
_
l
=
σ
(
W
s
l
h
l
[
i
]
+
b
s
l
)
p
l
e
_
l
=
σ
(
W
s
l
h
l
[
i
]
+
b
e
l
)
where l represents the predicted insertion position or the predicted deletion position,
p
i
s_l
represents a starting position,
p
i
e_l
represents an ending position, h l [i] is an encoded vector representation of an i-th word in the current utterance after passing through a transformer encoder,
W
s
1
and
W
e
1
represent weight vectors,
b
s
1
and
b
e
1
represent bias terms, and σ is a sigmoid activation function.
4 . The method according to claim 1 , wherein the semantic output variable can be represented as:
H
cln
=
λ
×
(
h
l
-
1
-
μ
h
)
/
(
σ
h
+
ε
)
+
β
where H cln is the semantic output variable;
h l-1 is an encoded vector obtained by encoding the current utterance by the transformer encoder;
λ=φ λ (H c )+λ, H c is the mean vector;
β=φ β (H c )+β, φ B and φ λ are two transformation matrices whose function is to transform the dimension of an input mean vector into a dimension consistent with λ and β;
H c =avg(h l [s], h l [e]), avg( ) represents a mean function, s represents a starting position, and e represents an ending position, and h l is an encoded vector obtained by encoding the input by the transformer encoder;
μ h and σ h are a mean value and a variance of feature vectors of different dimensions of the encoded vector corresponding to the current utterance at the current layer,
μ
h
=
1
/
H
∑
k
=
1
H
h
l
-
1
,
σ
h
=
1
/
H
∑
k
=
1
H
(
h
l
-
1
-
μ
h
)
2
,
H being a hidden layer dimension, and k being equal to 1 to H;
ε is a minimum.
5 . The method according to claim 1 , wherein the resolution completion sub-model determines the predicted position in the historical utterance using the following formula:
p
s
=
softmax
(
W
s
H
cln
+
b
s
)
p
e
=
softmax
(
W
s
H
cln
+
b
e
)
where p s is a starting position of the predicted position in the historical utterance, p e is an ending position of the predicted position in the historical utterance, softmax is a normalized exponential function, W s and W e represent weight vectors, b s and b e represent bias terms, and H cln is the semantic output variable.
6 . A method for training a coreference resolution model, comprising:
acquiring a plurality of conversation samples, wherein each of the conversation samples comprises a historical utterance, a current utterance and a reference utterance of the current utterance, and the reference utterance is an utterance after at least one of semantic deletion completion and coreference resolution of the current utterance;
acquiring annotation data of the conversation sample, wherein the annotation data comprises: at least one of a real insertion position where there is a semantic absence and a real deletion position of a word to be replaced in the current utterance, and at least one of a real position in the historical utterance of a word corresponding to the semantic absence at the real insertion position and a real position in the historical utterance of a replacement word corresponding to the word to be replaced at the real deletion position;
training, according to the current utterance and the annotation data of the conversation sample, a coreference resolution detection sub-model of the coreference resolution model to be trained to obtain a trained coreference resolution detection sub-model, wherein the trained coreference resolution detection sub-model is input with an utterance, and outputs at least one of a predicted insertion position where there is a semantic absence and a predicted deletion position of a word to be replaced in the input utterance;
training, according to the annotation data of the conversation sample and at least one of the predicted insertion position and the predicted deletion position of the current utterance of the conversation sample, a resolution completion sub-model of the coreference resolution model to be trained to obtain a trained resolution completion sub-model, wherein the trained resolution completion sub-model is input with at least one of a predicted insertion position and a predicted deletion position of the utterance, and outputs a predicted position in the historical utterance of the input utterance of a word corresponding to the semantic absence at at least one of the predicted insertion position and a predicted position in the historical utterance of the input utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position;
wherein training, according to the annotation data of the conversation sample and at least one of the predicted insertion position and the predicted deletion position of the current utterance of the conversation sample, a resolution completion sub-model of the coreference resolution model to be trained comprises:
a preprocessing step: calculating at least one of a mean vector of a starting position and an ending position of the predicted insertion position and a mean vector of a starting position and an ending position of the predicted deletion position of the current utterance; and taking the mean vector as a condition, performing semantic processing on an encoded vector obtained by encoding the current utterance by a transformer encoder by means of conditional layer normalization to obtain a conditional layer normalized semantic output variable; and
an input step: inputting the semantic output variable and the historical utterance of the conversation sample into the resolution completion sub-model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position; determining a value of a second objective function according to the predicted position and the real position in the historical utterance; and
if the value of the second objective function does not satisfy a second condition, adjusting parameters of the resolution completion sub-model, and returning to the input step until the value of the second objective function satisfies the second condition, so as to obtain the trained resolution completion sub-model.
7 . The method according to claim 6 , wherein the acquiring annotation data of the conversation sample comprises:
matching a sub-string in the current utterance of the conversation sample with a sub-string in the reference utterance to obtain a matching result;
if the matching result is that a sub-string only exists in the current utterance, determining that the sub-string is a word to be replaced, and a real deletion position of the word to be replaced in the current utterance, finding a replacement font corresponding to the word to be replaced in the reference utterance, and marking a real position in the historical utterance of a replacement word corresponding to the word to be replaced in the historical utterance at the real deletion position; and
if the matching result is that a sub-string only exists in the reference utterance, determining that there is a semantic absence in the current utterance, and determining a real insertion position where there is a semantic absence in the current utterance.
8 . The method according to claim 6 , wherein the training, according to the current utterance and the annotation data of the conversation sample, a coreference resolution detection sub-model of the coreference resolution model to be trained comprises:
an input step: inputting the current utterance of the conversation sample into the coreference resolution detection sub-model to obtain at least one of a predicted insertion position where there is a semantic absence of the current utterance and a predicted deletion position of a word to be replaced;
determining a value of a first objective function according to at least one of the predicted insertion position and the predicted deletion position output by the coreference resolution detection sub-model and at least one of a real insertion position and a real deletion position of the current utterance; and
if the value of the first objective function does not satisfy a first condition, adjusting parameters of the coreference resolution detection sub-model, and returning to the input step until the value of the first objective function satisfies the first condition, so as to obtain the trained coreference resolution detection sub-model.
9 . The method according to claim 8 , wherein the coreference resolution detection sub-model comprises two binary classification networks, wherein one binary classification network is input with the current utterance and outputs a 01 sequence for indicating whether the position of each word in the current utterance is a starting position or an ending position of the predicted insertion position, and the other binary classification network is input with the current utterance and outputs a 01 sequence for indicating whether the position of each word in the current utterance is a starting position or an ending position of the predicted deletion position.
10 . The method according to claim 9 , wherein the binary classification network uses the following formula to determine whether the position of each word in the current utterance is a starting position or an ending position of the predicted insertion position or the predicted deletion position:
p
l
s
_
l
=
σ
(
W
s
l
h
l
[
i
]
+
b
s
l
)
p
l
e
_
l
=
σ
(
W
s
l
h
l
[
i
]
+
b
e
l
)
where l represents the predicted insertion position or the predicted deletion position,
p
i
s
_
1
represents a starting position,
p
i
e
_
1
represents an ending position, h l [i] is an encoded vector representation of an i-th word in the current utterance after passing through a transformer encoder,
W
s
1
and
W
e
1
represent weight vectors to be trained,
b
s
1
and
b
e
1
represent bias terms to be trained, and σ is a sigmoid activation function.
11 . The method according to claim 8 , wherein the first objective function is expressed as follows:
loss
t
a
g
=
-
∑
i
=
1
n
+
2
y
i
s
_
l
log
(
p
i
s
_
l
)
-
∑
j
=
1
n
+
2
y
j
e
_
l
log
(
p
j
e
_
l
)
where loss tag is the first objective function, n is the length of the current utterance,
p
i
s
_
1
represents the starting position of the predicted insertion position or the predicted deletion position,
p
i
e
_
1
represents an ending position of the predicted insertion position or the predicted deletion position of the current utterance,
y
i
s
_
1
represents the starting position of the real insertion position or the real deletion position of the current utterance,
y
i
e
_
1
represents the ending position of the real insertion position or the real deletion position of the current utterance, i represents the i-th word in the current utterance, and j represents the j-th word in the current utterance.
12 . The method according to claim 6 , wherein the semantic output variable can be represented as:
H
cln
=
λ
×
(
h
l
-
1
-
μ
h
)
/
(
σ
h
+
ε
)
+
β
where H cln is the semantic output variable;
h l-1 is an encoded vector obtained by encoding the current utterance by the transformer encoder;
λ=φ λ (H c )+λ, H c is the mean vector;
β=φ β (H c )+β, φ β and φ λ are two transformation matrices whose function is to transform the dimension of an input mean vector into a dimension consistent with λ and β;
H c =avg(h l [s], h l [e]), avg( ) represents a mean function, s represents a starting position, and e represents an ending position, and h l is an encoded vector obtained by encoding the input by the transformer encoder;
μ h and σ h are a mean value and a variance of feature vectors of different dimensions of the encoded vector corresponding to the current utterance at the current layer,
μ
h
=
1
/
H
∑
k
=
1
H
h
l
-
1
,
σ
h
=
1
/
H
∑
k
=
1
H
(
h
l
-
1
-
μ
h
)
2
,
H being a hidden layer dimension, and k being equal to 1 to H;
ε is a minimum.
13 . The method according to claim 6 , wherein the resolution completion sub-model determines the predicted position in the historical utterance using the following formula:
p
s
=
softmax
(
W
s
H
cln
+
b
s
)
p
e
=
softmax
(
W
s
H
cln
+
b
e
)
where p s is a starting position of the predicted position in the historical utterance, p e is an ending position of the predicted position in the historical utterance, softmax is a normalized exponential function, W s and W e represent weight vectors to be trained, b s and b e represent bias terms to be trained, and H cln is the semantic output variable.
14 . The method according to claim 6 , wherein the second objective function is expressed as follows:
loss
mrc
=
-
∑
i
=
1
n
+
2
y
s
log
(
p
s
)
-
∑
j
=
1
n
+
2
y
e
log
(
p
e
)
where loss mrc is the second objective function, n is the length of the historical utterance, p s represents the starting position of the predicted position of the historical utterance, p e represents the ending position of the predicted position of the historical utterance, y s represents a starting position of the real position of the historical utterance, y e represents the ending position of the real position of the historical utterance, i represents the i-th word in the historical utterance, and j represents the j-th word in the historical utterance.
15 . An electronic device comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor which, when executed by the processor, to perform:
acquiring a current utterance to be processed;
inputting the current utterance into a coreference resolution detection sub-model of a coreference resolution model to obtain at least one of a predicted insertion position where there is a semantic absence of the current utterance and a predicted deletion position of a word to be replaced; and
inputting at least one of the predicted insertion position and the predicted deletion position of the current utterance, and a historical conversation corresponding to the current utterance into a resolution completion sub-model of the coreference resolution model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position;
wherein the inputting at least one of the predicted insertion position and the predicted deletion position of the current utterance, and a historical conversation corresponding to the current utterance into a resolution completion sub-model of the coreference resolution model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position, comprises:
calculating at least one of a mean vector of a starting position and an ending position of the predicted insertion position and a mean vector of a starting position and an ending position of the predicted deletion position of the current utterance;
taking the mean vector as a condition, performing semantic processing on an encoded vector obtained by encoding the current utterance by a transformer encoder by means of conditional layer normalization to obtain a conditional layer normalized semantic output variable; and
inputting the semantic output variable and the historical utterance of the current utterance into the resolution completion sub-model to obtain at least one of a predicted position in the historical utterance of a word corresponding to the semantic absence at the predicted insertion position and a predicted position in the historical utterance of a replacement word corresponding to the word to be replaced at the predicted deletion position.
16 . A non-transitory readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps of the coreference resolution method of claim 1 .
17 . An electronic device comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor which, when executed by the processor, carry out the steps of the method for training a coreference resolution model of claim 6 .
18 . A non-transitory readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps of the method for training a coreference resolution model of claim 6 .