IP Library › Granted Patent US 12,443,796
Granted Patent B2
US 12,443,796 · App. 17/815,630 · Granted Oct 14, 2025

Out of domain sentence detection

Inventors: Haode Qi (Cambridge, MA); Cheng Qian (Toronto, CA); Ladislav Kunc (Cambridge, MA); Saloni Potdar (Arlington, MA); Eric Donald Wayne (Raleigh, NC)
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
G06F40/279G06F16/3329G06F40/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,443,796
App. No.
17/815,630
Granted
Oct 14, 2025
Kind
B2
Abstract

A method, a structure, and a computer system for OOD sentence detection in dialogue systems. The exemplary embodiments may include receiving, for a domain corresponding to a particular topic, one or more on-topic text inputs and one or more off-topic text inputs. The exemplary embodiments may further include encoding the one or more on-topic text inputs and the one or more off-topic text inputs into a latent space, as well as decoding the one or more on-topic text inputs and the one or more off-topic text inputs from the latent space. The exemplary embodiments may additionally include minimizing a reconstruction error between the encoded one or more on-topic text inputs and the decoded one or more on-topic text inputs, and maximizing a reconstruction error between the encoded one or more off-topic text inputs and the decoded one or more off-topic text inputs.

Claims (76)

1. A computer-implemented method, comprising:

training, by at least one processor of a device, an autoencoder to detect user inputs that are out of a domain in which a dialogue system has been trained, wherein the training comprises:

receiving, for the domain of the dialogue system, training user input representations, wherein the training user input representations comprise respective weights based on respective importance of the training user input representations to the training, and wherein the training user input representations comprise in-domain training user input representations and out-of-domain training user input representations;

encoding the in-domain training user input representations into encoded in-domain training user input representations and the out-of-domain training user input representations into encoded out-of-domain training user input representations in a latent space;

decoding, from the latent space, the encoded in-domain training user input representations into decoded in-domain training user input representations and the encoded out-of-domain training user input representations into decoded out-of-domain training user input representations;

minimizing an in-domain reconstruction error between the in-domain training user input representations and the decoded in-domain training user input representations respectively corresponding to the in-domain training user input representations, wherein the in-domain reconstruction error is based on the respective weights of the in-domain training user input representations; and

maximizing an out-of-domain reconstruction error between the out-of-domain training user input representations and the decoded out-of-domain training user input representations respectively corresponding to the out-of-domain training user input representations, wherein the out-of-domain reconstruction error is based on the respective weights of the out-of-domain training user input representations;

receiving, by the at least one processor, using the dialogue system, a user input from a user;

determining, by the at least one processor, using the autoencoder, whether the user input is out of the domain; and

responsive to determining that the user input is out of the domain, directing the user to a different dialogue system trained for another domain that is different from the domain.

2. The computer-implemented method of claim 1 , wherein the encoding the in-domain training user input representations and the out-of-domain training user input representations into the latent space comprises:

converting, using at least one sentence encoder, the in-domain training user input representations into respective in-domain vectors;

converting, using the at least one sentence encoder, the out-of-domain training user input representations into respective out-of-domain vectors;

encoding, using the autoencoder, the respective in-domain vectors into respective in-domain latent inputs within the latent space; and

encoding, using the autoencoder, the respective out-of-domain vectors into one or more on topic latent inputs and one or more off topic respective out-of-domain latent inputs within the latent space.

3. The computer-implemented method of claim 2 , wherein the decoding the encoded in-domain training user input representations and the encoded out-of-domain training user input representations from the latent space comprises:

decoding, using the autoencoder, the respective in-domain latent inputs into the decoded in-domain training user input representations; and

decoding, using the autoencoder, the respective out-of-domain latent inputs into the decoded out-of-domain training user input representations.

4. The computer-implemented method of claim 1 , wherein the minimizing the in-domain reconstruction error comprises:

assigning a positive factor for a loss function between the encoded in-domain training user input representations and the decoded in-domain training user input representations.

5. The computer-implemented method of claim 1 , wherein the maximizing the out-of-domain reconstruction error comprises:

assigning a negative factor for a loss function between the encoded out-of-domain training user input representations and the decoded out-of-domain training user input representations.

6. The computer-implemented method of claim 1 , further comprising:

utilizing the in-domain reconstruction error and the out-of-domain reconstruction error as an indicator as to whether a user dialogue comprising the user input is in-domain or out-of-domain.

7. The computer-implemented method of claim 1 , wherein the training user input representations comprise previous user-provided inputs of the dialogue system.

8. A computer program product comprising a non-transitory computer-readable medium having instructions stored thereon that, in response to execution, cause a processor to perform operations comprising:

training an autoencoder to detect user inputs that are out of a domain in which a dialogue system has been trained, wherein the training comprises:

receiving, for the domain of the dialogue system, training user input representations, wherein the training user input representations comprise respective weights based on respective importance of the training user input representations to the training, and wherein the training user input representations comprise in-domain training user input representations and out-of-domain training user input representations;

encoding the in-domain training user input representations into encoded in-domain training user input representations and the out-of-domain training user input representations into encoded out-of-domain training user input representations in a latent space;

decoding, from the latent space, the encoded in-domain training user input representations into decoded in-domain training user input representations and the encoded out-of-domain training user input representations into decoded out-of-domain training user input representations;

minimizing an in-domain reconstruction error between the in-domain training user input representations and the decoded in-domain training user input representations respectively corresponding to the in-domain training user input representations, wherein the in-domain reconstruction error is based on the respective weights of the in-domain training user input representations; and

maximizing an out-of-domain reconstruction error between the out-of-domain training user input representations and the decoded out-of-domain training user input representations respectively corresponding to the out-of-domain training user input representations, wherein the out-of-domain reconstruction error is based on the respective weights of the out-of-domain training user input representations;

receiving, using the dialogue system, a user input from a user;

determining, using the autoencoder, whether the user input is out of the domain; and

responsive to determining that the user input is out of the domain, directing the user to a different dialogue system trained for another domain that is different from the domain.

9. The computer program product of claim 8 , wherein the encoding the in-domain training user input representations and the out-of-domain training user input representations into the latent space comprises:

converting, using at least one sentence encoder, the in-domain training user input representations into respective in-domain vectors;

converting, using the at least one sentence encoder, the out-of-domain training user input representations into respective out-of-domain vectors;

encoding, using the autoencoder, the respective in-domain vectors into respective in-domain latent inputs within the latent space; and

encoding, using the autoencoder, the respective out-of-domain vectors into respective out-of-domain latent inputs within the latent space.

10. The computer program product of claim 9 , wherein the decoding the encoded in-domain training user input representations and the encoded out-of-domain training user input representations from the latent space comprises:

decoding, using the autoencoder, the respective in-domain latent inputs into the decoded in-domain training user input representations; and

decoding, using the autoencoder, the respective out-of-domain latent inputs into the decoded out-of-domain training user input representations.

11. The computer program product of claim 8 , wherein the minimizing the in-domain reconstruction error comprises:

assigning a positive factor for a loss function between the encoded in-domain training user input representations and the decoded in-domain training user input representations.

12. The computer program product of claim 8 , wherein the maximizing the out-of-domain reconstruction error comprises:

assigning a negative factor for a loss function between the encoded out-of-domain training user input representations and the decoded out-of-domain training user input representations.

13. The computer program product of claim 8 , further comprising:

utilizing the in-domain reconstruction error and the out-of-domain reconstruction error as an indicator as to whether a user dialogue comprising the user input is in-domain or out-of-domain.

14. The computer program product of claim 8 , wherein the training user input representations comprise previous user-provided inputs of the dialogue system.

15. A system, comprising:

a memory configured to store computer executable instructions; and

a processor configured to execute at least one of the computer executable instructions that:

trains an autoencoder to detect user inputs that are out of a domain in which a dialogue system has been trained, wherein the training comprises:

receiving, for the domain of the dialogue system, training user input representations, wherein the training user input representations comprise respective weights based on respective importance of the training user input representations to the training, and wherein the training user input representations comprise in-domain training user input representations and out-of-domain training user input representations;

encoding the in-domain training user input representations into encoded in-domain training user input representations and the out-of-domain training user input representations into encoded out-of-domain training user input representations in a latent space;

decoding, from the latent space, the encoded in-domain training user input representations into decoded in-domain training user input representations and the encoded out-of-domain training user input representations into decoded out-of-domain training user input representations;

minimizing an in-domain reconstruction error between the in-domain training user input representations and the decoded in-domain training user input representations respectively corresponding to the in-domain training user input representations, wherein the in-domain reconstruction error is based on the respective weights of the in-domain training user input representations; and

maximizing an out-of-domain reconstruction error between the out-of-domain training user input representations and the decoded out-of-domain training user input representations respectively corresponding to the out-of-domain training user input representations, wherein the out-of-domain reconstruction error is based on the respective weights of the out-of-domain training user input representations;

receives, using the dialogue system, a user input from a user;

determines, using the autoencoder, whether the user input is out of the domain; and

responsive to determining that the user input is out of the domain, directs the user to a different dialogue system trained for another domain that is different from the domain.

16. The system of claim 15 , wherein the encoding the in-domain training user input representations and the out-of-domain training user input representations into the latent space comprises:

converting, using at least one sentence encoder, the in-domain training user input representations into respective in-domain vectors;

converting, using the at least one sentence encoder, the out-of-domain training user input representations into respective out-of-domain vectors;

encoding, using the autoencoder, the respective in-domain vectors into respective in-domain latent inputs within the latent space; and

encoding, using the autoencoder, the respective out-of-domain vectors into respective out-of-domain latent inputs within the latent space.

17. The system of claim 16 , wherein the decoding the encoded in-domain training user input representations and the encoded out-of-domain training user input representations from the latent space comprises:

decoding, using the autoencoder, the respective in-domain latent inputs into the decoded in-domain training user input representations; and

decoding, using the autoencoder, the respective out-of-domain latent inputs into the decoded out-of-domain training user input representations.

18. The system of claim 15 , wherein the minimizing the in-domain reconstruction error comprises:

assigning a positive factor for a loss function between the encoded in-domain training user input representations and the decoded in-domain training user input representations.

19. The system of claim 15 , wherein the maximizing the out-of-domain reconstruction error comprises:

assigning a negative factor for a loss function between the encoded out-of-domain training user input representations and the decoded out-of-domain training user input representations.

20. The system of claim 15 , further comprising:

utilizing the in-domain reconstruction error and the out-of-domain reconstruction error as an indicator as to whether a user dialogue comprising the user input is in-domain or out-of-domain.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 28, 2022
From: QI, HAODE; QIAN, CHENG; KUNC, LADISLAV; POTDAR, SALONI; WAYNE, ERIC DONALD
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 060655/0254 →
Continuity (1)
Related Publication 20240037331A1 · Feb 1, 2024
References Cited (16)
US 10528666B2 · Min · 2020 [cited by applicant]
US 10909442B1 · Szarvas · 2021 [cited by examiner]
US 11023683B2 · Padhi et al. · 2021 [cited by applicant]
US 20190050395A1 · Min · 2019 [cited by examiner]
US 20190171929A1 · Abadi · 2019 [cited by examiner]
US 20210034965A1 · Tan et al. · 2021 [cited by applicant]
US 20220230000A1 · Jalaluddin · 2022 [cited by examiner]
US 20220272054A1 · Gao · 2022 [cited by examiner]
An et al., “Variational Autoencoder based Anomaly Detection using Reconstruction Probability”, http://dm.snu.ac.kr/static/docs/TR/SNUDM-TR-2015-03.pdf, Dec. 27, 2015, pp. 1-18. [cited by applicant]
Disclosed Anonymously, “A Method to Identify Non-Contextual Words based on Word Weightage, Domain-Specific Glossary, Dictionary, and Thesaurus,” IPCOM000242682D, IP.com, Aug. 4, 2015, pp. 1-8. [cited by applicant]
IBM Cloud Docs, “Enhanced Irrelevance Detection Update”, Release notes for Watson Assistant, Mar. 1, 2022, https://cloud.ibm.com/docs/assistant?topic=assistant-release-notes, accessed Mar. 28, 2022, pp. 1-78. [cited by applicant]
Liang, et al., “GAN-Based Out-of-Domain Detection Using Both In-Domain and Out-of-Domain Samples,” ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 1-5. [cited by applicant]
Marek et al., “OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation”, https://arxiv.org/pdf/2104.02484.pdf, Apr. 6, 2021, pp. 1-8. [cited by applicant]
Mell et al., “The NIST Definition of Cloud Computing”, National Institute of Standards and Technology, Special Publication 800-145, Sep. 2011, pp. 1-7. [cited by applicant]
Ryu et al., “Neural Sentence Embedding Using only In-Domain Sentences for Out-of-Domain Sentence Detection in Dialog Systems,” Pattern Recognition Letters 88 (2017): pp. 1-8. [cited by applicant]
Zheng et al., “Out-of-domain Detection for Natural Language Understanding in Dialog Systems”, https://arxiv.org/pdf/1909.03862.pdf, Mar. 22, 2020, pp. 1-12. [cited by applicant]