Out-of-domain data augmentation for natural language processing
A method includes receiving a training set of utterances for training a machine-learning model to identify one or more intents for one or more utterances, and augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting includes: generating a data set of OOD examples, filtering out OOD examples from the data set of OOD examples, determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples, and generating augmented batches of utterances including utterances from the training set of utterances and utterances from the filtered data set of the OOD based on the difficulty value for each OOD. Thereafter, the machine-learning model is trained using the augmented batches of utterances in accordance with a curriculum training protocol.
1. A computer-implemented method comprising:
receiving a training set of utterances comprising in-domain examples;
augmenting the training set of utterances with out-of-domain (OOD) examples to generate augmented batches of utterances for training a machine-learning model, wherein the augmenting comprises:
generating a data set of the OOD examples,
filtering out a plurality of OOD examples from the data set of the OOD examples, based on a determination that context of each of the plurality of OOD examples has a substantial similarity to context of one or more of the utterances of the training set of utterances, and
generating the augmented batches of utterances, each of the augmented batches of utterances comprising utterances from the training set of utterances and utterances from the filtered data set of the OOD examples; and
training the machine-learning model using the augmented batches of utterances, wherein the trained machine-learning model is configured to, based on one or more utterances provided as an input by a user, identify an intent from a set of predetermined intents,
wherein the substantial similarity between the context of OOD utterances of the data set of the OOD examples and the context of the utterances of the training set is determined based on a distance measure using a Multilingual Universal Sentence Encoder (MUSE) single embedding, and
wherein if min (d_i) is less than a predetermined threshold, then the context of an OOD utterance of the data set of the OOD examples and the context of an utterance of the training set of utterances is determined to be substantially similar,
where d_i is an Euclidean distance (v_i, u),
v_i is a vector representation of an utterance (x_i) of the training set of utterances and is muse (x_i) where i=1→n, and
u is a vector representation of the OOD utterance of the data set of the OOD examples and is muse (OOD utterance).
2. The computer-implemented method of claim 1 , further comprising:
determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples.
3. The computer-implemented method of claim 2 , wherein the training the machine-learning model further comprises:
inputting the augmented batches of utterances to the machine-learning model based on the difficulty values of the OOD examples within each batch of the augmented batches of utterances.
4. The method of claim 3 , wherein the difficulty value is derived from the d_i value such that easy OOD examples will have a high d_i value, while hard OOD examples will have lower d_i values that are closer to the predetermined threshold.
5. The method of claim 4 , wherein the augmented batches of utterances are generated based on at least one from among (i) a predetermined batch size, (ii) incorporation of a predetermined number of the in-domain examples from the training set of utterances within each augmented batch, (iii) incorporation of a predetermined number of the OOD examples of utterances from the filtered data set of the OOD examples within each augmented batch, (iv) the predetermined number of the in-domain examples and the predetermined number of the OOD examples of utterances are selected to maintain the predetermined batch size, (v) the OOD examples of utterances incorporated into each augmented batch have a substantially similar difficulty value, and (vi) each OOD example from the filtered data set of the OOD examples is incorporated into a single batch of the augmented batches of utterances.
6. The method of claim 1 , further comprising deploying the trained machine-learning model in a chatbot system.
7. The method of claim 1 , wherein the augmented batches of utterances are generated using a batch balancing scheme.
8. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving a training set of utterances comprising in-domain examples;
augmenting the training set of utterances with out-of-domain (OOD) examples to generate augmented batches of utterances for training a machine-learning model, wherein the augmenting includes:
generating a data set of the OOD examples,
filtering out a plurality of OOD examples from the data set of the OOD examples, based on a determination that context of each of the plurality of OOD examples has a substantial similarity to context of one or more of the utterances of the training set of utterances, and
generating the augmented batches of utterances, each of the augmented batches of utterances comprising utterances from the training set of utterances and utterances from the filtered data set of the OOD examples; and
training the machine-learning model using the augmented batches of utterances, wherein the trained machine-learning model is configured to, based on one or more utterances provided as an input by a user, identify an intent from a set of predetermined intents,
wherein the substantial similarity between the context of OOD utterances of the data set of the OOD examples and the context of the utterances of the training set is determined based on a distance measure using a Multilingual Universal Sentence Encoder (MUSE) single embedding, and
wherein if min (d_i) is less than a predetermined threshold, then the context of an OOD utterance of the data set of the OOD examples and the context of an utterance of the training set of utterances is determined to be substantially similar,
where d_i is an Euclidean distance (v_i, u),
v_i is a vector representation of an utterance (x_i) of the training set of utterances and is muse (x_i) where i=1→n, and
u is a vector representation of the OOD utterance of the data set of the OOD examples and is muse (OOD utterance).
9. The system of claim 8 , wherein the operations further include:
determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples.
10. The system of claim 9 , wherein the training the machine-learning model further includes:
inputting the augmented batches of utterances to the machine-learning model based on the difficulty values of the OOD examples within each batch of the augmented batches of utterances.
11. The system of claim 10 , wherein the difficulty value is derived from the d_i value such that easy OOD examples will have a high d_i value, while hard OOD examples will have lower d_i values that are closer to the predetermined threshold.
12. The system of claim 11 , wherein the augmented batches of utterances are generated based on at least one from among (i) a predetermined batch size, (ii) incorporation of a predetermined number of the in-domain examples from the training set of utterances within each augmented batch, (iii) incorporation of a predetermined number of OOD examples of utterances from the filtered data set of the OOD examples within each augmented batch, (iv) the predetermined number of the in-domain examples and the predetermined number of OOD examples of utterances are selected to maintain the predetermined batch size, (v) OOD examples of utterances incorporated into each augmented batch have a substantially similar difficulty value, and (vi) each OOD example from the filtered data set of the OOD examples is incorporated into a single batch of the augmented batches of utterances.
13. The system of claim 8 , wherein the operations further include deploying the trained machine-learning model in a chatbot system.
14. The system of claim 8 , wherein the augmented batches of utterances are generated using a batch balancing scheme.
15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving a training set of utterances comprising in-domain examples;
augmenting the training set of utterances with out-of-domain (OOD) examples to generate augmented batches of utterances for training a machine-learning model, wherein the augmenting includes:
generating a data set of the OOD examples,
filtering out a plurality of OOD examples from the data set of the OOD examples, based on a determination that context of each of the plurality of OOD examples has a substantial similarity to context of one or more of the utterances of the training set of utterances, and
generating the augmented batches of utterances, each of the augmented batches of utterances comprising utterances from the training set of utterances and utterances from the filtered data set of the OOD examples; and
training the machine-learning model using the augmented batches of utterances, wherein the trained machine-learning model is configured to, based on one or more utterances provided as an input by a user, identify an intent from a set of predetermined intents,
wherein the substantial similarity between the context of OOD utterances of the data set of the OOD examples and the context of the utterances of the training set is determined based on a distance measure using a Multilingual Universal Sentence Encoder (MUSE) single embedding, and
wherein if min (d_i) is less than a predetermined threshold, then the context of an OOD utterance of the data set of the OOD examples and the context of an utterance of the training set of utterances is determined to be substantially similar,
where d_i is an Euclidean distance (v_i, u),
v_i is a vector representation of an utterance (x_i) of the training set of utterances and is muse (x_i) where i=1→n, and
u is a vector representation of the OOD utterance of the data set of the OOD examples and is muse (OOD utterance).
16. The non-transitory computer-readable memory of claim 15 , wherein the operations further include:
determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples.
17. The non-transitory computer-readable memory of claim 16 , wherein the training the machine-learning model further includes:
inputting the augmented batches of utterances to the machine-learning model based on the difficulty values of the OOD examples within each batch of the augmented batches of utterances.
18. The non-transitory computer-readable memory of claim 17 , wherein the difficulty value is derived from the d_i value such that easy OOD examples will have a high d_i value, while hard OOD examples will have lower d_i values that are closer to the predetermined threshold.
19. The non-transitory computer-readable memory of claim 18 , wherein the augmented batches of utterances are generated based on at least one from among (i) a predetermined batch size, (ii) incorporation of a predetermined number of the in-domain examples from the training set of utterances within each augmented batch, (iii) incorporation of a predetermined number of OOD examples of utterances from the filtered data set of the OOD examples within each augmented batch, (iv) the predetermined number of the in-domain examples and the predetermined number of OOD examples of utterances are selected to maintain the predetermined batch size, (v) OOD examples of utterances incorporated into each augmented batch have a substantially similar difficulty value, and (vi) each OOD example from the filtered data set of the OOD examples is incorporated into a single batch of the augmented batches of utterances.
20. The non-transitory computer-readable memory of claim 15 , wherein the operations further include deploying the trained machine-learning model in a chatbot system.