Prompt augmented generative replay via supervised contrastive training for lifelong intent detection
Embodiments disclosed herein model lifelong intent detection as a class-incremental learning where a new set of intents/classes are added at each incremental step. To address the issue of catastrophic forgetting during lifelong intent detection (LID), an incremental learner is provided with Prompt Augmented Generative Replay, wherein unlike existing approaches that store real samples in replay memory, only concept words obtained from old intents are stored, which reduces memory consumption and speeds up incremental training still enabling not forgetting the old intents. Joint training of an incremental learner is carried out for LID and a pseudo-labeled utterance generation with objective is to classify a user utterance into one of multiple pre-defined intents by minimizing a total Loss function comprising a LID loss function, a Labeled Utterance Generation loss function, a Supervised Contrastive Training loss function, and a Knowledge Distillation loss function.
1 . A processor implemented method for Lifelong Intent Detection (LID), the method comprising:
performing a joint training, by one or more hardware processor, of an incremental learner implemented via the one or more hardware processors, to obtain a trained incremental learner for LID and a pseudo-labeled utterance generation, wherein the pseudo-labeled utterances refer to synthetic labeled data comprising utterance and intent, wherein the utterance is generated based on prompt based approach, wherein the joint training repeats at regular intervals for new input intent is received, and wherein the plurality of concept words from the new input intent are identified as top-K term frequency-inverse document frequency (tf-idf)) words, wherein a LID problem is modeled as a class Incremental Learning (class IL) problem, and wherein a training objective is to classify a user utterance into one of multiple pre-defined intents by minimizing a total Loss function (L total ), wherein the total Loss function (L total ) comprises:
a) a Lifelong Intent Detection (LID) loss function (L ID ) to train the incremental learner for an intent label generation, wherein the incremental learner generates the intent label corresponding to an input utterance using a prompt-based generative classification approach, wherein no overlap between the intents are generated at different incremental steps, wherein a prompt is created using one of a Prompt without question (PWQ) technique and a Prompt without incremental question (PWIQ) technique;
b) a Labeled Utterance Generation (LUG) loss function (L R ) to train the incremental learner for the pseudo-labeled utterance generation, wherein the incremental learner generates the pseudo-labeled utterance for an input intent label of an input prompt using a prompt augmented generative replay approach, wherein the input prompt is created using a LUG prompt function and comprises an input intent and a plurality of concept words from the input intent stored in a memory, wherein only storing relevant contexts per intent in the memory and using stored contexts with the class label as prompts for generating intent-specific utterances, thereby eliminating need of volumes of old training data, and reducing memory requirement, speeding up incremental training still enabling not forgetting old intents;
c) a Supervised Contrastive Training (SCT) loss function (L SCT ) to fine-tune the incremental learner using a valid utterance intent pair (U, I1) and a randomly chosen intent (I2) to predict whether I1 and I2 correspond to a same intent, wherein to minimize the likelihood of incorrect utterance-label pairs during LID and LUG, the trained incremental learner is explicitly trained on positive and negative utterance-label pairs via contrastive loss; and
d) a Knowledge Distillation (KD) loss function (L KD ) to fine-tune the incremental learner to alleviate catastrophic forgetting;
generating, by the trained incremental learner of a previous instance, implemented by the one or more hardware processors, a current set of pseudo-labeled utterances;
receiving, at a current instance, labeled data corresponding to new input intents by the one or more hardware processor;
iterating, by the one or more hardware processors, the joint training of the trained incremental learner of previous instance using the current set of pseudo-labeled utterances and the current instance labeled data corresponding to new input intents to generate a trained incremental learner for current instance; and
identifying, by the one or more hardware processor, a current plurality of concept words from the current instance labeled data corresponding to new input intents and storing the plurality of concept words in a replay memory for generation of the pseudo-labeled utterances in successive joint training of incremental learner.
2 . The processor implemented method of claim 1 , wherein the total Loss function (L total ) is mathematically represented as: L total =λ1*L ID +λ1*L KD +λ2+L R +λ3*L SCT , wherein λ1, λ2, λ3 are set of hyperparameters of the incremental learner.
3 . The processor implemented method of claim 1 , wherein the prompt created using the PWQ technique is based on a PWQ prompt function, mathematically represented as:
f
prompt
PWQ
(
x
)
=
(
x
,
ANS
,
y
,
EOS
)
,
wherein x is the user utterance, ANS and EOS refers to special tokens used during prompt creation, and y is labeled data.
4 . The processor implemented method of claim 1 , wherein the prompt created using the PWIQ technique is based on a PWIQ prompt function, mathematically represented as:
f
prompt
PWIQ
(
x
)
=
(
x
,
IncQ
,
ANS
,
y
,
EOS
)
,
wherein x is the user utterance, ANS, EOS refers to special tokens used during prompt creation, and IncQ at a current instance includes all the intent labels from y labeled data.
5 . A system for Lifelong Intent Detection (LID), the system comprising:
a memory storing instructions;
one or more Input/Output (I/O) interfaces; and
one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
perform a joint training of an incremental learner implemented via the one or more hardware processors, to obtain a trained incremental learner for LID and a pseudo-labeled utterance generation, wherein the pseudo-labeled utterances refer to synthetic labeled data comprising utterance and intent, wherein the utterance is generated based on prompt based approach, wherein the joint training repeats at regular intervals for new input intent is received, and wherein the plurality of concept words from the new input intent are identified as top-K term frequency-inverse document frequency (tf-idf)) words, wherein a LID problem is modeled as a class Incremental Learning (class IL) problem, and wherein a training objective is to classify a user utterance into one of multiple pre-defined intents by minimizing a total Loss function (L total ), wherein the total Loss function (L total ) comprises:
a) a Lifelong Intent Detection (LID) loss function (L ID ) to train the incremental learner for an intent label generation, wherein the incremental learner generates the intent label corresponding to an input utterance using a prompt-based generative classification approach, wherein no overlap between the intents are generated at different incremental steps, wherein a prompt is created using one of a Prompt without question (PWQ) technique and a Prompt without incremental question (PWIQ) technique;
b) a Labeled Utterance Generation (LUG) loss function (L R ) to train the incremental learner for the pseudo-labeled utterance generation, wherein the incremental learner generates the pseudo-labeled utterance for an input intent label of an input prompt using a prompt augmented generative replay approach, wherein the input prompt is created using a LUG prompt function and comprises an input intent and a plurality of concept words from the input intent stored in a memory, wherein only storing relevant contexts per intent in the memory and using stored contexts with the class label as prompts for generating intent-specific utterances, thereby eliminating need of volumes of old training data, and reducing memory requirement, speeding up incremental training still enabling not forgetting old intents;
c) a Supervised Contrastive Training (SCT) loss function (L SCT ) to finetune the incremental learner using a valid utterance intent pair (U, I1) and a randomly chosen intent (I2) to predict whether I1 and I2 correspond to a same intent, wherein to minimize the likelihood of incorrect utterance-label pairs during LID and LUG, the trained incremental learner is explicitly trained on positive and negative utterance-label pairs via contrastive loss; and
d) a Knowledge Distillation (KD) loss function (L KD ) to fine-tune the incremental learner to alleviate catastrophic forgetting;
generate via the trained incremental learner of a previous instance, a current set of pseudo-labeled utterances;
receive, at a current instance, labeled data corresponding to new input intents;
iterate the joint training of the trained incremental learner of previous instance using the current set of pseudo-labeled utterances and the current instance labeled data corresponding to new input intents to generate a trained incremental learner for current instance; and
identify a current plurality of concept words from the current instance labeled data corresponding to new input intents and storing the plurality of concept words in a replay memory for generation of the pseudo-labeled utterances in successive joint training of incremental learner.
6 . The system of claim 5 , wherein the total Loss function (L) is mathematically represented as: L total =λ1*L ID +λ1×L KD +λ2*L R +λ3*L SCT , wherein λ1, λ2, λ3 are set of hyperparameters of the incremental learner.
7 . The system of claim 5 , wherein the prompt created using the PWQ technique is based on a PWQ prompt function, mathematically represented as:
f
prompt
PWQ
(
x
)
=
(
x
,
ANS
,
y
,
EOS
)
,
wherein x is the user utterance, ANS and EOS refers to special tokens used during prompt creation, and y is labeled data.
8 . The system of claim 5 , wherein the prompt created using the PWIQ technique is based on a PWIQ prompt function, mathematically represented as:
f
prompt
PWIQ
(
x
)
=
(
x
,
IncQ
,
ANS
,
y
,
EOS
)
,
wherein x is the user utterance, ANS, EOS refers to special tokens used during prompt creation, and IncQ at a current instance includes all the intent labels from y labeled data.
9 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
performing a joint training of an incremental learner implemented via the one or more hardware processors, to obtain a trained incremental learner for LID and a pseudo-labeled utterance generation, wherein the pseudo-labeled utterances refer to synthetic labeled data comprising utterance and intent, wherein the utterance is generated based on prompt based approach, wherein the joint training repeats at regular intervals for new input intent is received, and wherein the plurality of concept words from the new input intent are identified as top-K term frequency-inverse document frequency (tf-idf)) words, wherein a LID problem is modeled as a class Incremental Learning (class IL) problem, and wherein a training objective is to classify a user utterance into one of multiple pre-defined intents by minimizing a total Loss function (L total ), wherein the total Loss function (L total ) comprises:
a) a Lifelong Intent Detection (LID) loss function (L ID ) to train the incremental learner for an intent label generation, wherein the incremental learner generates the intent label corresponding to an input utterance using a prompt-based generative classification approach, wherein no overlap between the intents are generated at different incremental steps, wherein a prompt is created using one of a Prompt without question (PWQ) technique and a Prompt without incremental question (PWIQ) technique;
b) a Labeled Utterance Generation (LUG) loss function (L R ) to train the incremental learner for the pseudo-labeled utterance generation, wherein the incremental learner generates the pseudo-labeled utterance for an input intent label of an input prompt using a prompt augmented generative replay approach, wherein the input prompt is created using a LUG prompt function and comprises an input intent and a plurality of concept words from the input intent stored in a memory wherein only storing relevant contexts per intent in the memory and using stored contexts with the class label as prompts for generating intent-specific utterances, thereby eliminating need of volumes of old training data, and reducing memory requirement, speeding up incremental training still enabling not forgetting old intents;
c) a Supervised Contrastive Training (SCT) loss function (L SCT ) to fine-tune the incremental learner using a valid utterance intent pair (U, I1) and a randomly chosen intent (I2) to predict whether I1 and I2 correspond to a same intent, wherein to minimize the likelihood of incorrect utterance-label pairs during LID and LUG, the trained incremental learner is explicitly trained on positive and negative utterance-label pairs via contrastive loss; and
d) a Knowledge Distillation (KD) loss function (L KD ) to fine-tune the incremental learner to alleviate catastrophic forgetting;
generating by the trained incremental learner of a previous instance, implemented by the one or more hardware processors, a current set of pseudo-labeled utterances;
receiving at a current instance, labeled data corresponding to new input intents by the one or more hardware processor;
iterating by the one or more hardware processors, the joint training of the trained incremental learner of previous instance using the current set of pseudo-labeled utterances and the current instance labeled data corresponding to new input intents to generate a trained incremental learner for current instance; and
identifying by the one or more hardware processor, a current plurality of concept words from the current instance labeled data corresponding to new input intents and storing the plurality of concept words in a replay memory for generation of the pseudo-labeled utterances in successive joint training of incremental learner.
10 . The one or more non-transitory machine-readable information storage mediums of claim 9 , wherein the total Loss function (L total ) is mathematically represented as: L total =λ1*L ID +Δ1*L KD +λ2*L R +λ3*L SCT , wherein λ1, λ2, λ3 are set of hyperparameters of the incremental learner.
11 . The one or more non-transitory machine-readable information storage mediums of claim 9 , wherein the prompt created using the PWQ technique is based on a PWQ prompt function, mathematically represented as:
f
prompt
PWQ
(
x
)
=
(
x
,
ANS
,
y
,
EOS
)
,
wherein x is the user utterance, ANS and EOS refers to special tokens used during prompt creation, and y is labeled data.
12 . The one or more non-transitory machine-readable information storage mediums of claim 9 wherein the prompt created using the PWIQ technique is based on a PWIQ prompt function, mathematically represented as:
f
prompt
PWIQ
(
x
)
=
(
x
,
IncQ
,
ANS
,
y
,
EOS
)
,
wherein x is the user utterance, ANS, EOS refers to special tokens used during prompt creation, and IncQ at a current instance includes all the intent labels from y labeled data.