Automatically assessing question answering system performance across possible confidence values

1. A method, in a data processing system having a processor and a memory, wherein the memory comprises instructions which are executed by the processor to cause the processor to implement a performance assessment system for assessing question answering system performance, the method comprising:

receiving, by the performance assessment system, question answering system results, wherein the question answering system results comprise questions posed to the question answering system, answers returned by the question answering system for each question posed to the question answering system, and a confidence value for each answer, and wherein the question answering system is trained or tested using the ground truth questions and answers;

performing, by a data manipulation and calculation component executing within the performance assessment system, a matching operation comparing each question in the question answering system results to questions in the ground truth, wherein a given question is determined to be on-topic or off-topic based on results of the matching operation;

for a plurality of confidence threshold values for an answer confidence threshold, determining, by the data manipulation and calculation component, a rightness or wrongness s of each answer in the question answering system results;

generating, by the performance assessment system, performance statistics for the plurality of confidence threshold values based on whether each question is on-topic or off-topic and whether each answer is right or wrong based on the plurality of confidence threshold values for the answer confidence threshold;

generating, by a web interface component executing within the performance assessment system, a threshold control panel graphical user interface (GUI) comprising a current threshold control for setting a current answer confidence threshold value;

presenting, by the web interface component, the threshold control panel GUI to a user;

responsive to the user interacting with the current answer confidence control to set the current answer confidence threshold value, presenting, by the web interface component, the performance statistics to the user via a user interface, wherein the performance statistics comprise an on-topic correct answer probability value and an off-topic incorrect answer probability value corresponding to the current answer confidence threshold value; and

configuring the question answering system with the current answer confidence threshold value.

2. The method of claim 1 , wherein receiving question answering system results comprises submitting each question from the ground truth questions and answers to the question answering system and receiving a response answer from the question answering system for each submitted question and a confidence value associated with each response answer.

3. The method of claim 1 , wherein receiving question answering system results comprises receiving question answering system log results via a question answering system log application programming interface.

4. The method of claim 1 , wherein determining a rightness or wrongness of each answer in the question answering system results comprises performing a matching operation comparing each given answer in the question answering system results to a corresponding ground truth answer.

5. The method of claim 1 , wherein the performance statistics further comprise a hybrid Receiver Operating Characteristics (ROC) curve that plots probability of a true answer versus probability of a false answer for the plurality of confidence threshold values.

6. The method of claim 5 , wherein presenting the performance statistics comprises:

presenting the hybrid ROC curve; and

marking a location on the hybrid ROC curve corresponding to the current answer confidence threshold value.

7. The method of claim 1 , wherein the performance statistics further comprise a correctness curve that plots precision versus questions answered for the plurality of confidence threshold values.

8. The method of claim 7 , wherein presenting the performance statistics comprises:

presenting the correctness curve; and

marking a location on the correctness curve corresponding to the current answer confidence threshold value.

9. The method of claim 1 , wherein the performance statistics further comprise an on-topic precision value and an on-topic attempted percentage value corresponding to the current answer confidence threshold value.

10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a performance assessment system for assessing question answering system performance, wherein the computer readable program causes the computing device to:

receive, by the performance assessment system, question answering system results, wherein the question answering system results comprise questions posed to the question answering system, answers returned by the question answering system for each question posed to the question answering system, and a confidence value for each answer, and wherein the question answering system is trained or tested using the ground truth questions and answers;

perform, by a data manipulation and calculation component executing within the performance assessment system, a matching operation comparing each question in the question answering system results to questions in the ground truth, wherein a given question is determined to be on-topic or off-topic based on results of the matching operation;

for a plurality of confidence threshold values for an answer confidence threshold, determine, by the data manipulation and calculation component, a rightness or wrongness of each answer in the question answering system results;

generate, by the performance assessment system, performance statistics for the plurality of confidence threshold values based on whether each question is on-topic or off-topic and whether each answer is right or wrong based on the plurality of confidence threshold values for the answer confidence threshold;

generate, by a web interface component executing within the performance assessment system, a threshold control panel graphical user interface (GUI) comprising a current threshold control for setting a current answer confidence threshold value;

present, by the web interface component, the threshold control panel GUI to a user;

responsive to the user interacting with the current answer confidence control to set the current answer confidence threshold value, present, by the web interface component, the performance statistics to the user via a user interface, wherein the performance statistics comprise an off-topic incorrect answer probability value corresponding to the current answer confidence threshold value; and

configure the question answering system with the current answer confidence threshold value.

11. The computer program product of claim 10 , wherein receiving question answering system results comprises submitting each question from the ground truth questions and answers to the question answering system and receiving a response answer from the question answering system for each submitted question and a confidence value associated with each response answer.

12. The computer program product of claim 10 , wherein receiving question answering system results comprises receiving question answering system log results via a question answering system log application programming interface.

13. The computer program product of claim 10 , wherein determining a rightness or wrongness of each answer in the question answering system results comprises performing a matching operation comparing each given answer in the question answering system results to a corresponding ground truth answer.

14. The computer program product of claim 10 , wherein the performance statistics further comprise a hybrid Receiver Operating Characteristics (ROC) curve that plots probability of a true answer versus probability of a false answer for the plurality of confidence threshold values.

15. The computer program product of claim 14 , wherein presenting the performance statistics comprises:

presenting the hybrid ROC curve; and

marking a location on the hybrid ROC curve corresponding to the current answer confidence threshold value.

16. The computer program product of claim 10 , wherein the performance statistics further comprise a correctness curve that plots precision versus questions answered for the plurality of confidence threshold values.

17. The computer program product of claim 16 , wherein presenting the performance statistics comprises:

presenting the correctness curve; and

marking a location on the correctness curve corresponding to the current answer confidence threshold value.

18. The computer program product of claim 10 , wherein the performance statistics further comprise an on-topic precision value and an on-topic attempted percentage value corresponding to the current answer confidence threshold value.

19. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a performance assessment system for assessing question answering system performance, wherein the instructions further cause the processor to:

receive, by the performance assessment system, question answering system results, wherein the question answering system results comprise questions posed to the question answering system, answers returned by the question answering system for each question posed to the question answering system, and a confidence value for each answer, and wherein the question answering system is trained or tested using the ground truth questions and answers;

perform, by a data manipulation and calculation component executing within the performance assessment system, a matching operation comparing each question in the question answering system results to questions in the ground truth, wherein a given question is determined to be on-topic or off-topic based on results of the matching operation;

for a plurality of confidence threshold values for an answer confidence threshold, determine, by the data manipulation and calculation component, a rightness or wrongness of each answer in the question answering system results;

generate, by the performance assessment system, performance statistics for the plurality of confidence threshold values based on whether each question is on-topic or off-topic and whether each answer is right or wrong based on the plurality of confidence threshold values for the answer confidence, threshold;

generate, by a web interface component executing within the performance assessment system, a threshold control panel graphical user interface (GUI) comprising a current threshold control for setting a current answer confidence threshold value;

present, by the web interface component, the threshold control panel GUI to a user;

responsive to the user interacting with the current answer confidence control to set the current answer confidence threshold value, present, by the web interface component, the performance statistics to the user via a user interface, wherein the performance statistics comprise an off-topic incorrect answer probability value corresponding to the current answer confidence threshold value; and

configure the question answering system with the current answer confidence threshold value.

20. The apparatus of claim 19 , wherein determining a rightness or wrongness of each answer in the question answering system results comprises performing a matching operation comparing each given answer in the question answering system results to a corresponding ground truth answer.

21. The apparatus of claim 19 , wherein the performance statistics further comprise an on-topic precision value and an on-topic attempted percentage value corresponding to the current answer confidence threshold value.