Determining off-topic questions in a question answering system using probabilistic language models
A question answering system that determines whether a question is off-topic by performing the following steps: (i) receiving, by a question answering system, a set of documents; (ii) identifying topical subset(s) for each document of the set of documents using named entity recognition, where each topical subset relates to a corresponding topic; (iii) assigning a set of topic score(s) for each topical subset using natural language processing, where each topic score relates to a corresponding probability associated with the respective topical subset under a probabilistic language model; and (iv) determining, based, at least in part, on the topic score(s) corresponding to the topical subset(s), whether a question input into the question answering system is off-topic.
1. A computer-implemented method comprising:
receiving, by a question answering system adapted to answer questions for a particular domain, an input question;
determining: (i) a first logarithm probability associated with the input question under an on-topic probabilistic language model for the particular domain, and (ii) a second logarithm probability associated with the input question under an off-topic probabilistic language model for the particular domain;
modelling, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability;
assigning, based, at least in part, on the modelled difference, a topic score to the input question; and
determining, based, at least in part, on the assigned topic score, whether the input question is off-topic with respect to the particular domain.
2. The method of claim 1 , wherein the determining of whether the input question is off-topic with respect to the particular domain includes comparing the assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the input question.
3. The method of claim 2 , further comprising determining the input question is off-topic based, at least in part, on the assigned topic score being greater than the associated relevance score for the generated answer.
4. The method of claim 2 , further comprising determining the input question is on-topic based, at least in part, on the assigned topic score being less than the associated relevance score for the generated answer.
5. The method of claim 1 , further comprising:
responsive to determining that the input question is off-topic, informing a user who input the input question into the question answering system that the input question is off-topic.
6. The method of claim 1 , wherein the on-topic probabilistic language model is trained using questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is trained using questions that have been identified as being off-topic with respect to the particular domain.
7. A computer program product comprising a computer readable storage medium having stored thereon:
program instructions to receive, by a question answering system adapted to answer questions for a particular domain, an input question;
program instructions to determine: (i) a first logarithm probability associated with the input question under an on-topic probabilistic language model for the particular domain, and (ii) a second logarithm probability associated with the input question under an off-topic probabilistic language model for the particular domain;
program instructions to model, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability;
program instructions to assign, based, at least in part, on the modelled difference, a topic score to the input question; and
program instructions to determine, based, at least in part, on the assigned topic score, whether the input question is off-topic with respect to the particular domain.
8. The computer program product of claim 7 , wherein determining whether the input question is off-topic with respect to the particular domain includes comparing the assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the input question.
9. The computer program product of claim 8 , the computer readable storage medium having further stored thereon program instructions to determine the input question is off-topic based, at least in part, on the assigned topic score being greater than the associated relevance score for the generated answer.
10. The computer program product of claim 8 , the computer readable storage medium having further stored thereon program instructions to determine the input question is on-topic based, at least in part, on the assigned topic score being less than the associated relevance score for the generated answer.
11. The computer program product of claim 7 , the computer readable storage medium having further stored thereon program instructions to, responsive to determining that the input question is off-topic, inform a user who input the input question into the question answering system that the input question is off-topic.
12. The computer program product of claim 7 , wherein the on-topic probabilistic language model is trained using questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is trained using questions that have been identified as being off-topic with respect to the particular domain.
13. A computer system comprising:
a processor(s) set; and
a computer readable storage medium;
wherein:
the processor set is structured, located, connected and/or programmed to run program instructions stored on the computer readable storage medium; and
the stored program instructions include:
program instructions to receive, by a question answering system adapted to answer questions for a particular domain, an input question;
program instructions to determine: (i) a first logarithm probability associated with the input question under an on-topic probabilistic language model for the particular domain, and (ii) a second logarithm probability associated with the input question under an off-topic probabilistic language model for the particular domain;
program instructions to model, using a sigmoid function, a difference between the first logarithm probability and the second logarithm probability;
program instructions to assign, based, at least in part, on the modelled difference, a topic score to the input question; and
program instructions to determine, based, at least in part, on the assigned topic score, whether the input question is off-topic with respect to the particular domain.
14. The computer system of claim 13 , wherein determining whether the input question is off-topic with respect to the particular domain includes comparing the assigned topic score to a relevance score associated with an answer generated by the question answering system in response to the input question.
15. The computer system of claim 14 , the stored program instructions further including program instructions to determine the input question is off-topic based, at least in part, on the assigned topic score being greater than the associated relevance score for the generated answer.
16. The computer system of claim 14 , the stored program instructions further including program instructions to determine the input question is on-topic based, at least in part, on the assigned topic score being less than the associated relevance score for the generated answer.
17. The computer system of claim 13 , the stored program instructions further including program instructions to, responsive to determining that the input question is off-topic, inform a user who input the input question into the question answering system that the input question is off-topic.
18. The computer system of claim 13 , wherein the on-topic probabilistic language model is trained using questions that have been identified as being on-topic with respect to the particular domain, and wherein the off-topic probabilistic language model is trained using questions that have been identified as being off-topic with respect to the particular domain.