Hierarchical context tagging for utterance rewriting
Hierarchical context tagging for utterance rewriting comprising computer code for obtaining source tokens and context tokens, encoding the source tokens and the context tokens to generate source contextualized embeddings and context contextualized embeddings, tagging the source tokens with tags indicating a keep or delete action for each source token of the source tokens, selecting a rule to insert before the each source token, wherein the rule contains a sequence of one or more slots, and generating spans from the context tokens, wherein each span corresponds to one of the one or more slots of the selected rule.
1 . A method performed by at least one processor of a multi-span tagger (MST) of hierarchical context tagging for utterance rewriting, the method comprising:
obtaining source tokens and context tokens;
encoding, using a Bidirectional Encoder Representations from Transforms (BERT) model, the source tokens and the context tokens to generate first source contextualized embeddings and first context contextualized embeddings;
tagging, using the first source contextualized embeddings, the source tokens with tags indicating a keep or delete action for each source token of the source tokens;
selecting, using the first context contextualized embeddings, a rule containing a sequence of one or more slots to insert before the each source token;
generating spans from the context tokens, each span corresponding to one of the one or more slots of the selected rule; and
performing, using the spans, utterance rewriting on a multi-turn dialogue to recover one or more coreferences on a latest turn of the multi-turn dialogue.
2 . The method of claim 1 , wherein the source tokens and the context tokens are concatenated before encoding.
3 . The method of claim 1 , further comprising:
adding a predetermined token to the beginning of the source tokens and the context tokens; and
encoding the source tokens and the context tokens, with the predetermined token added, to generate second source contextualized embeddings and second context contextualized embeddings,
wherein the second source contextualized embeddings and second context contextualized embeddings are used to represent the rule.
4 . The method of claim 1 , wherein the source tokens are tagged by linearly projecting a corresponding source contextualized embedding using a learned parameter matrix.
5 . The method of claim 1 , wherein the rule is selected by linearly projecting a corresponding source contextualized embedding using a rule classifier.
6 . The method of claim 1 , wherein the sequence of one or more slots are non-terminals that are only rewritten as terminals from the generated spans; and
wherein a predetermined number of the one and more slots are filled.
7 . The method of claim 1 , wherein the spans are generated autoregressively, and a current span is dependent on all previous spans for a corresponding source token.
8 . The method of claim 1 , further comprising generating a special slot token to represent slots at a same position across rules.
9 . The method of claim 1 , wherein a deleted source token is replaced with the generated spans.
10 . An apparatus for utterance rewriting using hierarchical context tagging, the apparatus comprising:
at least one memory configured to store computer program code;
at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code including:
first obtaining code configured to cause the at least one processor to obtain source tokens and context tokens;
first encoding code configured to cause the at least one processor to encode, using a Bidirectional Encoder Representations from Transforms (BERT) model, the source tokens and the context tokens to generate first source contextualized embeddings and first context contextualized embeddings;
first tagging code configured to cause the at least one processor to tag, using the first source contextualized embeddings, the source tokens with tags indicating a keep or delete action for each source token of the source tokens;
first selecting code configured to cause the at least one processor to select, using the first context contextualized embeddings, a rule containing a sequence of one or more slots to insert before the each source token;
first generating code configured to cause the at least one processor to generate spans from the context tokens, each span corresponding to one of the one or more slots of the selected rule; and
performing code configured to cause the at least one processor to perform, using the spans, utterance rewriting on a multi-turn dialogue to recover one or more coreferences on a latest turn of the multi-turn dialogue.
11 . The apparatus of claim 10 , wherein the source tokens and the context tokens are concatenated before encoding.
12 . The apparatus of claim 10 , further comprising:
concatenating code configured to cause the at least one processor to add a predetermined token to the beginning of the source tokens and the context tokens; and
second encoding code configured to cause the at least one processor to encode the source tokens and the context tokens, with the added predetermined token, to generate second source contextualized embeddings and second context contextualized embeddings,
wherein the second source contextualized embeddings and second context contextualized embeddings are used to represent the rule.
13 . The apparatus of claim 10 , wherein the source tokens are tagged by linearly projecting a corresponding source contextualized embedding using a learned parameter matrix, and the rule is selected by linearly projecting a corresponding source contextualized embedding using a rule classifier.
14 . The apparatus of claim 10 , wherein the spans are generated autoregressively, and a current span is dependent on all previous spans for a corresponding source token.
15 . The apparatus of claim 10 , further comprising second generating code configured to cause the at least one processor to generate a special slot token to represent slots at a same position across rules.
16 . The apparatus of claim 10 , wherein a deleted source token is replaced with the generated spans.
17 . A non-transitory computer readable medium storing instructions, that when executed by at least one processor of a multi-span tagger (MST), cause the at least one processor to:
obtain source tokens and context tokens;
encode, using a Bidirectional Encoder Representations from Transforms (BERT) model, the source tokens and the context tokens to generate first source contextualized embeddings and first context contextualized embeddings;
tag, using the first source contextualized embeddings, the source tokens with tags indicating a keep or delete action for each source token;
select, using the first context contextualized embeddings, a rule containing a sequence of one or more slots, to insert before the each source token;
generate spans from the context tokens, each span corresponding to one of the one or more slots of the selected rule; and
performing, using the spans, utterance rewriting on a multi-turn dialogue to recover one or more coreferences on a latest turn of the multi-form dialogue.
18 . The non-transitory computer-readable medium of claim 17 , wherein the source tokens and the context tokens are concatenated before encoding.
19 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to:
add a predetermined token to the beginning of the source tokens and the context tokens; and
encode the source tokens and the context tokens, with the added predetermined token, to generate second source contextualized embeddings and second context contextualized embeddings,
wherein the second source contextualized embeddings and second context contextualized embeddings are used to represent the rule.
20 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to generate a special slot token to represent slots at a same relative position across rules.