Unsupervised dynamic confidence thresholding for answering questions

1. A method, comprising:

receiving, by a question answering system having a confidence threshold, plural questions from one or more user devices; and

processing, by the question answering system, each respective one of the plural questions by:

generating an answer to the respective one of the plural questions;

determining a confidence score of the answer;

in response to determining the confidence score is greater than the confidence threshold, increasing the confidence threshold and returning the answer to a respective one of the one or more user devices that generated the respective one of the plural questions; and

in response to determining the confidence score is less than the confidence threshold, decreasing the confidence threshold and not returning the answer to the respective one of the one or more user devices that generated the respective one of the plural questions,

wherein the increasing the confidence threshold and the decreasing the confidence threshold are performed such that the question answering system returns answers for the plural questions at a rate that approximates a pre-defined target answering frequency.

2. The method of claim 1 , further comprising receiving input defining the target answering frequency.

3. The method of claim 2 , wherein the input is received from one of the one or more user devices.

4. The method of claim 2 , wherein the increasing the confidence threshold and the decreasing the confidence threshold are performed using at least one algorithm that is a function of the target answering frequency and a learning rate.

5. The method of claim 4 , further comprising receiving input defining the learning rate.

6. The method of claim 4 , further comprising adjusting the learning rate, during the processing, until the learning rate reaches a predefined minimum value.

7. The method of claim 1 , wherein the plural questions comprise:

a first set of questions comprising seed questions that cause the question answering system to adjust the confidence threshold from an initial value toward a value that approximates the target answering frequency; and

a second set of questions from users in a target audience.

8. The method of claim 7 , wherein the first set of questions are presented to the question answering system prior to the second set of questions being presented to the question answering system.

9. A computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to:

receive, by a question answering system, a question from a user device;

generate, by the question answering system, an answer to the question;

determine, by the question answering system, a confidence score of the answer;

compare, by the question answering system, the confidence score to a confidence threshold; and

adjust the confidence threshold, by the question answering system and based on the comparing, by performing one of:

in response to the confidence score being greater than the confidence threshold, increasing the confidence threshold for a next question; and

in response to the confidence score being less than the confidence threshold, decreasing the confidence threshold for the next question;

wherein the increasing the confidence threshold and the decreasing the confidence threshold are performed using at least one algorithm that is a function of a target answering frequency and a learning rate.

10. The computer program product of claim 9 , wherein the program instructions are executable to:

return the answer to the user device in response to the confidence score being greater than the confidence threshold; and

not return the answer to the user device in response to the confidence score being less than the confidence threshold.

11. The computer program product of claim 9 , wherein the program instructions are executable to repeat the receiving, the generating, the determining, the comparing, and the adjusting for plural different questions presented to the question answering system.

12. The computer program product of claim 11 , wherein the adjusting the confidence threshold during the repeating causes the question answering system to approximate the target answering frequency for the plural different questions.

13. The computer program product of claim 12 , wherein the program instructions are executable to receive user input defining the target answering frequency.

14. The computer program product of claim 11 , wherein the program instructions are executable to adjust the learning rate, during the repeating, until the learning rate reaches a predefined minimum value.

15. A question answering system comprising:

a processor, a computer readable memory, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to:

receive input defining a target answering frequency of the question answering system;

receive a question from a user device;

generate one or more answers to the question;

determine a respective confidence score for each of the one or more answers;

determine which of the one or more answers to return to the user device based on a confidence threshold; and

adjust the confidence threshold based on comparing a highest one of the respective confidence scores to the confidence threshold, wherein the adjusting the confidence threshold is performed in a manner that causes the question answering system to approximate the target answering frequency.

16. The question answering system of claim 15 , wherein:

the one or more answers comprise plural answers;

the program instructions are executable to determine a respective discounted confidence score for each of the plural answers; and

the determining which of the one or more answers to return to the user device comprises comparing the respective discounted confidence scores to the confidence threshold.

17. The question answering system of claim 16 , wherein each of the respective discounted confidence scores comprises an independence discounted confidence score.

18. The question answering system of claim 16 , wherein each of the respective discounted confidence scores comprises a redundancy discounted confidence score.

19. The question answering system of claim 15 , wherein the adjusting the confidence threshold comprises:

in response to the highest one of the respective confidence scores being greater than the confidence threshold, incrementally increasing the confidence threshold using a first algorithm so that the question answering system is less likely to return an answer for a next question; and

in response to the highest one of the respective confidence scores being less than the confidence threshold, incrementally decreasing the confidence threshold using a second algorithm so that the question answering system is more likely to return an answer for the next question.

20. The method of claim 1 , wherein the plural questions comprise:

a first set of questions comprising seed questions; and

a second set of questions from users in a target audience.