System and method for automated analysis of ground truth using confidence model to prioritize correction options
A method, system and computer-usable medium are disclosed for automated analysis of ground truth using confidence model to prioritize correction options. In certain embodiments, the ground truth data is analyzed to identify review-candidates. A confidence level may be assigned to each of the identified review-candidates and the review-candidates are prioritized, at least in part, using the assigned confidence levels. The review-candidates are electronically presented in prioritized order to solicit verification or correction feedback for updating the ground truth data.
1. A computer-implemented method for automated analysis of ground truth using an information processing system having a processor and a memory, the method comprising:
receiving, by the information processing system, ground truth data;
analyzing, by the information processing system, the ground truth data to identify review-candidates;
assigning, by the information processing system, a confidence level to each of the identified review-candidates;
prioritizing, by the information processing system, the review-candidates based at least on the assigned confidence levels;
electronically presenting, by the information processing system, the review-candidates in prioritized order to solicit corrective feedback for updating the ground truth data;
generating, by the information processing system, suggested fixes for the review-candidates; and
grouping identified review candidates having the same suggested fixes;
electronically presenting the grouped review-candidates in prioritized order along with the suggested fixes to solicit corrective feedback for updating the ground truth data using the suggested fixes; and,
training a question answer (QA) system using the suggested fixes.
2. The computer-implemented method of claim 1 , wherein prioritizing the review-candidates further comprises:
prioritizing a review-candidate based on an impact of changing the review-candidate in the ground truth data using one or more of the respective suggested fixes.
3. The computer-implemented method of claim 2 , wherein
the impact of changing the review-candidate in the ground truth data is based, at least in part, on a number of ground truth data entries that would be changed using the respective suggested fixes.
4. The computer-implemented method of claim 1 , further comprising:
identifying, by the information processing system, review-candidates based on similarities between different attribute names; and
assigning, by the information processing system, a high confidence level to review-candidates having different attribute names within a predetermined edit distance.
5. The computer-implemented method of claim 1 , further comprising:
identifying, by the information processing system, review-candidates based on differences in data types in ground truth entries for a given attribute; and
assigning, by the information processing system, a high confidence level to review-candidates having different data types for the given attribute.
6. A system comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
receiving ground truth data;
analyzing the ground truth data to identify review-candidates;
assigning a confidence level to each of the identified review-candidates;
prioritizing the review-candidates based at least on the assigned confidence levels;
electronically presenting the review-candidates in prioritized order to solicit corrective feedback for updating the ground truth data;
generating, by the information processing system, suggested fixes for the review-candidates; and
grouping identified review candidates having the same suggested fixes;
electronically presenting the grouped review-candidates in prioritized order along with the suggested fixes to solicit corrective feedback for updating the ground truth data using the suggested fixes; and,
training a question answer (QA) system using the suggested fixes.
7. The system of claim 6 , wherein prioritizing the review-candidates further comprises:
prioritizing a review-candidate based on an impact of changing the review-candidate in the ground truth data using one or more of the respective suggested fixes.
8. The system of claim 7 , wherein:
the impact of changing the review-candidate in the ground truth data is based, at least in part, on a number of ground truth data entries that would be changed using the respective suggested fixes.
9. The system of claim 6 , wherein the instructions are further configured for:
identifying review-candidates based on similarities between different attribute names; and
assigning a high confidence level to review-candidates having different attribute names within a predetermined edit distance.
10. The system of claim 6 , wherein the instructions are further configured for:
identifying review-candidates based on differences in data types in ground truth entries for a given attribute; and
assigning a high confidence level to review-candidates having different data types for the given attribute.
11. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
receiving ground truth data;
analyzing the ground truth data to identify review-candidates;
assigning a confidence level to each of the identified review-candidates;
prioritizing the review-candidates based at least on the assigned confidence levels;
electronically presenting the review-candidates in prioritized order to solicit corrective feedback for updating the ground truth data;
generating, by the information processing system, suggested fixes for the review-candidates; and
grouping identified review candidates having the same suggested fixes;
electronically presenting the grouped review-candidates in prioritized order along with the suggested fixes to solicit corrective feedback for updating the ground truth data using the suggested fixes; and,
training a question answer (QA) system using the suggested fixes.
12. The non-transitory, computer-readable storage medium of claim 11 , wherein prioritizing the review-candidates further comprises:
prioritizing a review-candidate based on an impact of changing the review-candidate in the ground truth data using one or more of the respective suggested fixes.
13. The non-transitory, computer-readable storage medium of claim 12 , wherein
the impact of changing the review-candidate in the ground truth data is based, at least in part, on a number of ground truth data entries that would be changed using the respective suggested fixes.
14. The non-transitory, computer-readable storage medium of claim 11 , wherein the instructions are further configured for:
identifying review-candidates based on similarities between different attribute names; and
assigning a high confidence level to review-candidates having different attribute names within a predetermined edit distance.
15. The non-transitory, computer-readable storage medium of claim 11 , wherein the instructions are further configured for:
identifying review-candidates based on differences in data types in ground truth entries for a given attribute; and
assigning a high confidence level to review-candidates having different data types for the given attribute.