Techniques for out-of-domain (OOD) detection
The present disclosure relates to techniques for identifying out-of-domain utterances. One particular technique includes receiving an utterance and a target domain of a chatbot, generating a sentence embedding for the utterance, obtaining an embedding representation for each cluster of in-domain utterances associated with the target domain, predicting, using a metric learning model, a first probability that the utterance belongs to the target domain based on a similarity or difference between the sentence embedding and each embedding representation for each cluster, predicting, using an outlier detection model, a second probability that the utterance belongs to the target domain based on a determined distance or density deviation between the sentence embedding and embedding representations for neighboring clusters, evaluating the first probability and the second probability to determine a final probability, and classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.
1. A computer-implemented method comprising:
accessing an utterance and a target domain of a chatbot;
generating a sentence embedding for the utterance;
predicting a first probability as to whether the utterance belongs to the target domain of the chatbot based on the sentence embedding for the utterance and a distance or density deviation between the sentence embedding and an embedding representation for a cluster of a plurality of clusters of in-domain utterances associated with the target domain of the chatbot;
predicting a second probability as to whether the utterance belongs to the target domain of the chatbot based on the sentence embedding for the utterance and a similarity or difference between the sentence embedding and an embedding representation for a cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot;
determining, based on the first probability and the second probability, a final probability as to whether the utterance belongs to the target domain of the chatbot; and
classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.
2. The method of claim 1 , wherein an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot is an average of sentence embeddings for each in-domain utterance in the given cluster.
3. The method of claim 1 , wherein the first probability is predicted by inputting the sentence embedding for the utterance and an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot into an outlier detection model constructed with a distance or density algorithm for outlier detection.
4. The method of claim 3 , wherein the outlier detection model is configured to calculate the distance or density deviation.
5. The method of claim 1 , wherein the second probability is predicted by inputting the sentence embedding for the utterance and an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot into a metric learning model having learned model parameters configured to provide a probability as to whether the utterance belongs to the target domain.
6. The method of claim 5 , wherein the metric learning model is configured to calculate the similarity or difference.
7. The method of claim 1 , wherein the sentence embedding for the utterance is generated using an embedding model that maps natural language elements including sentences, words, and n-grams into arrays of numbers, and wherein each natural language element of the natural language elements is represented as a single point in a vector space.
8. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
accessing an utterance and a target domain of a chatbot;
generating a sentence embedding for the utterance;
predicting a first probability as to whether the utterance belongs to the target domain of the chatbot based on the sentence embedding for the utterance and a distance or density deviation between the sentence embedding and an embedding representation for a cluster of a plurality of clusters of in-domain utterances associated with the target domain of the chatbot;
predicting a second probability as to whether the utterance belongs to the target domain of the chatbot based on the sentence embedding for the utterance and a similarity or difference between the sentence embedding and an embedding representation for a cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot;
determining, based on the first probability and the second probability, a final probability as to whether the utterance belongs to the target domain of the chatbot; and
classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.
9. The computer-program product of claim 8 , wherein an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot is an average of sentence embeddings for each in-domain utterance in the given cluster.
10. The computer-program product of claim 8 , wherein the first probability is predicted by inputting the sentence embedding for the utterance and an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot into an outlier detection model constructed with a distance or density algorithm for outlier detection.
11. The computer-program product of claim 10 , wherein the outlier detection model is configured to calculate the distance or density deviation.
12. The computer-program product of claim 8 , wherein the second probability is predicted by inputting the sentence embedding for the utterance and an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot into a metric learning model having learned model parameters configured to provide a probability as to whether the utterance belongs to the target domain.
13. The computer-program product of claim 12 , wherein the metric learning model is configured to calculate the similarity or difference.
14. The computer-program product of claim 8 , wherein the sentence embedding for the utterance is generated using an embedding model that maps natural language elements including sentences, words, and n-grams into arrays of numbers, and wherein each natural language element of the natural language elements is represented as a single point in a vector space.
15. A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
accessing an utterance and a target domain of a chatbot;
generating a sentence embedding for the utterance;
predicting a first probability as to whether the utterance belongs to the target domain of the chatbot based on the sentence embedding for the utterance and a distance or density deviation between the sentence embedding and an embedding representation for a cluster of a plurality of clusters of in-domain utterances associated with the target domain of the chatbot;
predicting a second probability as to whether the utterance belongs to the target domain of the chatbot based on the sentence embedding for the utterance and a similarity or difference between the sentence embedding and an embedding representation for a cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot;
determining, based on the first probability and the second probability, a final probability as to whether the utterance belongs to the target domain of the chatbot; and
classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.
16. The system of claim 15 , wherein an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot is an average of sentence embeddings for each in-domain utterance in the given cluster.
17. The system of claim 15 , wherein the first probability is predicted by inputting the sentence embedding for the utterance and an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot into an outlier detection model constructed with a distance or density algorithm for outlier detection.
18. The system of claim 17 , wherein the outlier detection model is configured to calculate the distance or density deviation.
19. The system of claim 15 , wherein the second probability is predicted by inputting the sentence embedding for the utterance and an embedding representation for a given cluster of the plurality of clusters of in-domain utterances associated with the target domain of the chatbot into a metric learning model having learned model parameters configured to provide a probability as to whether the utterance belongs to the target domain.
20. The system of claim 19 , wherein the metric learning model is configured to calculate the similarity or difference.