Techniques for out-of-domain (OOD) detection
The present disclosure relates to techniques for identifying out-of-domain utterances. One particular technique includes receiving an utterance and a target domain of a chatbot, generating a sentence embedding for the utterance, obtaining an embedding representation for each cluster of in-domain utterances associated with the target domain, predicting, using a metric learning model, a first probability that the utterance belongs to the target domain based on a similarity or difference between the sentence embedding and each embedding representation for each cluster, predicting, using an outlier detection model, a second probability that the utterance belongs to the target domain based on a determined distance or density deviation between the sentence embedding and embedding representations for neighboring clusters, evaluating the first probability and the second probability to determine a final probability, and classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.
1. A computer-implemented method comprising:
generating an embedding for an input utterance;
predicting, based on the embedding and a distance or density algorithm, a first probability as to whether the input utterance belongs to a target domain of a chatbot;
predicting, based on the embedding and a deep learning network, a second probability as to whether the input utterance belongs to the target domain of the chatbot;
determining, based on the first probability and the second probability, a final probability as to whether the input utterance belongs to the target domain of the chatbot; and
classifying, based on the first probability and the second probability, the input utterance is an in-domain utterance or out-of-domain utterance for the chatbot.
2. The method of claim 1 , wherein the first probability is predicted based on a distance or density deviation between the embedding and embedding representations for clusters of in-domain utterances associated with the target domain.
3. The method of claim 2 , wherein each embedding representation of the embedding representations for the clusters is associated with a respective cluster of the clusters.
4. The method of claim 2 , wherein the embedding representations for the clusters are obtained by generating a sentence embedding for each in-domain utterance of the in-domain utterances and inputting the sentence embedding for each in-domain utterance into an unsupervised clustering model configured to identify clusters.
5. The method of claim 1 , wherein the second probability is predicted based on a similarity or difference between the embedding and embedding representations for clusters of in-domain utterances associated with the target domain.
6. The method of claim 5 , wherein each embedding representation of the embedding representations for the clusters is associated with a respective cluster of the clusters.
7. The method of claim 1 , wherein the deep learning network comprises at least one of a stacked highway network and a combination of a linear model and a deep neural network.
8. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
generating an embedding for an input utterance;
predicting, based on the embedding and a distance or density algorithm, a first probability as to whether the input utterance belongs to a target domain of a chatbot;
predicting, based on the embedding and a deep learning network, a second probability as to whether the input utterance belongs to the target domain of the chatbot;
determining, based on the first probability and the second probability, a final probability as to whether the input utterance belongs to the target domain of the chatbot; and
classifying, based on the first probability and the second probability, the input utterance is an in-domain utterance or out-of-domain utterance for the chatbot.
9. The computer-program product of claim 8 , wherein the first probability is predicted based on a distance or density deviation between the embedding and embedding representations for clusters of in-domain utterances associated with the target domain.
10. The computer-program product of claim 9 , wherein each embedding representation of the embedding representations for the clusters is associated with a respective cluster of the clusters.
11. The computer-program product of claim 9 , wherein the embedding representations for the clusters are obtained by generating a sentence embedding for each in-domain utterance of the in-domain utterances and inputting the sentence embedding for each in-domain utterance into an unsupervised clustering model configured to identify clusters.
12. The computer-program product of claim 8 , wherein the second probability is predicted based on a similarity or difference between the embedding and embedding representations for clusters of in-domain utterances associated with the target domain.
13. The computer-program product of claim 12 , wherein each embedding representation of the embedding representations for the clusters is associated with a respective cluster of the clusters.
14. The computer-program product of claim 8 , wherein the deep learning network comprises at least one of a stacked highway network and a combination of a linear model and a deep neural network.
15. A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
generating an embedding for an input utterance;
predicting, based on the embedding and a distance or density algorithm, a first probability as to whether the input utterance belongs to a target domain of a chatbot;
predicting, based on the embedding and a deep learning network, a second probability as to whether the input utterance belongs to the target domain of the chatbot;
determining, based on the first probability and the second probability, a final probability as to whether the input utterance belongs to the target domain of the chatbot; and
classifying, based on the first probability and the second probability, the input utterance is an in-domain utterance or out-of-domain utterance for the chatbot.
16. The system of claim 15 , wherein the first probability is predicted based on a distance or density deviation between the embedding and embedding representations for clusters of in-domain utterances associated with the target domain.
17. The system of claim 16 , wherein each embedding representation of the embedding representations for the clusters is associated with a respective cluster of the clusters.
18. The system of claim 16 , wherein the embedding representations for the clusters are obtained by generating a sentence embedding for each in-domain utterance of the in-domain utterances and inputting the sentence embedding for each in-domain utterance into an unsupervised clustering model configured to identify clusters.
19. The system of claim 15 , wherein the second probability is predicted based on a similarity or difference between the embedding and embedding representations for clusters of in-domain utterances associated with the target domain.
20. The system of claim 19 , wherein each embedding representation of the embedding representations for the clusters is associated with a respective cluster of the clusters.