Systems and methods for response selection in multi-party conversations with dynamic topic tracking

1. A system for dynamic topic tracking in a multi-party conversation involving interactions between at least two user participants and the system on different topics, the system comprising:

a memory configured to store a language model, a context history of a plurality of prior utterances from multiple user participants relating to multiple topics, and a plurality of candidate responses at a current turn of the multi-party conversation;

a processor configured to:

input, to the language model, each prior utterance from the context history paired with a candidate response from the plurality of candidate responses;

encode, via the language model, pairs of the prior utterances and the candidate response into a plurality of topic vectors;

generate, by a self-attention layer, a plurality of self-attended topic vectors indicative of topic relevance at an utterance level by inputting and letting the plurality of topic vectors attend to each other at the utterance level;

compute a relevance score for the candidate response given the context history based on max-pooling of the plurality of attended topic vectors; and

determine whether to select the candidate response as a response at the current turn of the multi-party conversation base on the relevance score.

2. The system of claim 1 , wherein the processor is further configured to encode, via the language model, pairs of the prior utterances and the candidate response into the plurality of topic vectors by:

generating an input sequence of tokens representing a pair of an utterance and a candidate response;

encoding, via a transformer layer in the language model, the input sequence of tokens into an encoded representation including a first portion representing a start token in the input sequence of tokens and a second portion representing remaining tokens in the input sequence of tokens;

attending over the second portion of the encoded representation based on the first portion as query; and

concatenating the attended second portion of the encoded representation and the first portion to result in a topic vector from the plurality of topic vectors.

3. The system of claim 1 , wherein the processor is further configured to compute the relevance score for the candidate response given the context history based on max-pooling of the plurality of attended topic vectors by:

generating a max-pooling output from the plurality of attended topic vectors;

performing a softmax operation over a linear mapping of the max-pooling output to obtain the relevance score,

wherein the relevance score indicates a relevance level between the respective prior utterance and the candidate response; and

computing a first entropy loss based on the relevance score and a ground truth label.

4. The system of claim 3 , wherein the processor is further configured to:

compute a binary topic classifier based on the plurality of topic vectors, wherein the binary topic classifier indicates whether the respective prior utterance and the candidate response belongs to a same topic; and

compute a second cross entropy loss based on the binary topic classifier.

5. The system of claim 4 , where the processor is further configured to:

compute a reply-to distribution based on the plurality of attended topic vectors, wherein the reply-to distribution indicates a probability that the candidate response replies to the respective prior utterance; and

compute a third cross-entropy loss based on the reply-to distribution.

6. The system of claim 5 , wherein the processor is further configured to:

compute a combined loss as a weighted sum of the first cross-entropy loss, the second cross-entropy loss and the third cross-entropy loss; and

jointly update a response selection module, a topic prediction module and a topic entanglement module based on the combined loss.

7. The system of claim 1 , wherein the language model is pre-trained with a pretraining dataset including a plurality of utterances, each utterance being paired with a respective positive response from the multi-party conversation and a respective negative response from outside the multi-party conversation.

8. The system of claim 7 , wherein the processor is further configured to:

input a first utterance and a second utterance, from the pretraining dataset, in a form of a training sequence to the language model, wherein the training sequence includes a first token that predicts whether the first utterance and the second utterance belong to a same topic;

generate, by embedding, a token representation of the training sequence;

generate, by an encoder layer of the language model, encoded topic vectors of the token representation, wherein the encoded topic vector includes a first encoded topic vector corresponding to the first token and wherein the first encoded topic vector encodes a topic relationship between the first utterance and the second utterance;

determine whether the first utterance and the second utterance are matched in topic using the encoded first token as a contextual embedding; and

update the language model using a determined topic relationship between the first utterance and the second utterance.

9. The system of claim 8 , wherein the processor is further configured to:

mask at least a portion of the pretraining dataset comprising the plurality of utterances, paired positive responses and paired negative responses; and

train the language model using the masked pretraining dataset based on a masked language modeling loss.

10. The system of claim 8 , wherein the token representation includes a first representation corresponding to a start token in the training sequence, and the processor is further configured to:

encode, via a transformer layer in the language model, a topic relationship between the first utterance and the second utterance into the first representation.

11. A method for dynamic topic tracking in a multi-party conversation involving interactions between at least two user participants and the system on different topics, the method comprising:

retrieving, from a memory, a language model, a context history of a plurality of prior utterances from multiple user participants relating to multiple topics, and a plurality of candidate responses at a current turn of the multi-party conversation;

inputting, to the language model, each prior utterance from the context history paired with a candidate response from the plurality of candidate responses;

encoding, via the language model, pairs of the prior utterances and the candidate response into a plurality of topic vectors;

generating, by a self-attention layer, a plurality of self-attended topic vectors indicative of topic relevance at an utterance level by inputting and letting the plurality of topic vectors attend to each other at the utterance level;

computing a relevance score for the candidate response given the context history based on max-pooling of the plurality of attended topic vectors; and

determining whether to select the candidate response as a response at the current turn of the multi-party conversation base on the relevance score.

12. The method of claim 11 , wherein the encoding, via the language model, pairs of the prior utterances and the candidate response into the plurality of topic vectors comprises:

generating an input sequence of tokens representing a pair of an utterance and a candidate response;

encoding, via a transformer layer in the language model, the input sequence of tokens into an encoded representation including a first portion representing a start token in the input sequence of tokens and a second portion representing remaining tokens in the input sequence of tokens;

attending over the second portion of the encoded representation based on the first portion as query; and

concatenating the attended second portion of the encoded representation and the first portion to result in a topic vector from the plurality of topic vectors.

13. The method of claim 11 , wherein the computing the relevance score for the candidate response given the context history based on max-pooling of the plurality of attended topic vectors comprises:

generating a max-pooling output from the plurality of attended topic vectors;

performing a softmax operation over a linear mapping of the max-pooling output to obtain the relevance score,

wherein the relevance score indicates a relevance level between the respective prior utterance and the candidate response; and

computing a first entropy loss based on the relevance score and a ground truth label.

14. The method of claim 13 , further comprising:

computing a binary topic classifier based on the plurality of topic vectors, wherein the binary topic classifier indicates whether the respective prior utterance and the candidate response belongs to a same topic; and

computing a second cross entropy loss based on the binary topic classifier.

15. The method of claim 14 , further comprising:

computing a reply-to distribution based on the plurality of attended topic vectors, wherein the reply-to distribution indicates a probability that the candidate response replies to the respective prior utterance; and

computing a third cross-entropy loss based on the reply-to distribution.

16. The method of claim 15 , further comprising:

computing a combined loss as a weighted sum of the first cross-entropy loss, the second cross-entropy loss and the third cross-entropy loss; and

jointly updating a response selection module, a topic prediction module and a topic entanglement module based on the combined loss.

17. The method of claim 11 , wherein the language model is pre-trained with a pretraining dataset including a plurality of utterances, each utterance being paired with a respective positive response from the multi-party conversation and a respective negative response from outside the multi-party conversation.

18. The method of claim 17 , further comprising:

inputting a first utterance and a second utterance, from the pretraining dataset, in a form of a training sequence to the language model, wherein the training sequence includes a first token that predicts whether the first utterance and the second utterance belong to a same topic;

generating, by embedding, a token representation of the training sequence;

generating, by an encoder layer of the language model, encoded topic vectors of the token representation, wherein the encoded topic vector includes a first encoded topic vector corresponding to the first token and wherein the first encoded topic vector encodes a topic relationship between the first utterance and the second utterance;

determining whether the first utterance and the second utterance are matched in topic using the encoded first token as a contextual embedding; and

updating the language model using a determined topic relationship between the first utterance and the second utterance.

19. The method of claim 18 , wherein the token representation includes a first representation corresponding to a start token in the training sequence, and the method further comprises:

encoding, via a transformer layer in the language model, a topic relationship between the first utterance and the second utterance into the first representation.

20. A processor-readable non-transitory storage medium storing processor-executable instructions for dynamic topic tracking in a multi-party conversation involving interactions between at least two user participants and the system on different topics, the processor-executable instructions being executed by a processor to perform operations including:

retrieving a language model, a context history of a plurality of prior utterances from multiple user participants relating to multiple topics, and a plurality of candidate responses at a current turn of the multi-party conversation;

inputting, to the language model, each prior utterance from the context history paired with a candidate response from the plurality of candidate responses;

encoding, via the language model, pairs of the prior utterances and the candidate response into a plurality of topic vectors;

generating, by a self-attention layer, a plurality of self-attended topic vectors indicative of topic relevance at an utterance level by inputting and letting the plurality of topic vectors attend to each other at the utterance level;

computing a relevance score for the candidate response given the context history based on max-pooling of the plurality of attended topic vectors; and

determining whether to select the candidate response as a response at the current turn of the multi-party conversation base on the relevance score.