Statistical determination of operator error
Various embodiments of the invention provide systems and methods for estimating a frequency of errors for data processing operators, but without human inspection of documents typed by the operators and without comparing documents typed by the operators against known transcriptions.
1. A computer-implemented method for estimating a frequency of errors for each of a plurality of data processing operators, the data processing operators performing a data processing operation on a plurality of documents, the method comprising a computer system executing software to effect the steps of:
assigning documents to different groups of operators, each group containing at least two operators, each operator belonging to at least two groups;
making the documents that are assigned to a group, available to the operators in the group in order for each operator to type data in the documents assigned to the group, the different operators thus generating different versions for each document;
collecting the versions of the documents typed by the different operators;
statistically analyzing discrepancies between the different versions without human intervention and without comparing to a known transcription of the document; and
based on the statistical analysis of documents assigned to groups and based on the assignment of operators to groups, estimating an average frequency of errors for each operator without knowing a frequency of errors for any specific document typed by that operator, the frequency of errors averaged over the plurality of documents.
2. The computer-implemented method of claim 1 wherein the step of assigning documents to different groups of operators comprises randomly assigning documents to different groups of operators.
3. The computer-implemented method of claim 1 wherein each group of operators consists of a pair of operators.
4. The computer-implemented method of claim 1 wherein the step of estimating an average frequency of errors for each operator comprises:
estimating an average frequency of errors for each group of operators, based on the statistical analysis of documents assigned to that group of operators; and
based on the assignment of operators to groups and the estimated average frequency of errors for each group, estimating the average frequency of errors for each operator.
5. The computer-implemented method of claim 1 wherein:
the documents comprise fields; and
the step of estimating an average frequency of errors for each operator comprises estimating an average frequency of errors for at least one field for each operator.
6. The computer-implemented method of claim 1 wherein the step of estimating an average frequency of errors comprises estimating an average frequency of errors for each field.
7. A computer program product for a quality management system, the computer program product stored on a tangible computer-readable medium and including instructions that, when loaded into memory, cause a processor to carry out the steps of:
assigning documents to different groups of operators, each group containing at least two operators, each operator belonging to at least two groups;
making the documents that are assigned to a group, available to the operators in the group in order for each operator to type data in the documents assigned to the group, the different operators thus generating different versions for each document;
collecting the versions of the documents typed by the different operators;
statistically analyzing discrepancies between the different versions without human intervention and without comparing to a known transcription of the document; and
based on the statistical analysis of documents assigned to groups and based on the assignment of operators to groups, estimating an average frequency of errors for each operator without knowing a frequency of errors for any specific document typed by that operator, the frequency of errors averaged over the plurality of documents.
8. The computer program product of claim 7 wherein the step of assigning documents to different groups of operators comprises randomly assigning documents to different groups of operators.
9. The computer program product of claim 7 wherein each group of operators consists of a pair of operators.
10. The computer program product of claim 7 wherein the step of estimating an average frequency of errors for each operator comprises:
estimating an average frequency of errors for each group of operators, based on the statistical analysis of documents assigned to that group of operators; and
based on the assignment of operators to groups and the estimated average frequency of errors for each group, estimating the average frequency of errors for each operator.
11. The computer program product of claim 7 wherein:
the documents comprise fields; and
the step of estimating an average frequency of errors for each operator comprises estimating an average frequency of errors for at least one field for each operator.
12. The computer program product of claim 7 wherein the step of estimating an average frequency of errors comprises estimating an average frequency of errors for each field.
13. A quality management system comprising:
means for assigning documents to different groups of operators, each group containing at least two operators, each operator belonging to at least two groups;
means for making the documents that are assigned to a group, available to the operators in the group in order for each operator to type data in the documents assigned to the group, the different operators thus generating different versions for each document;
means for collecting the versions of the documents typed by the different operators;
means for statistically analyzing discrepancies between the different versions without human intervention and without comparing to a known transcription of the document; and
means for, based on the statistical analysis of documents assigned to groups and based on the assignment of operators to groups, estimating an average frequency of errors for each operator without knowing a frequency of errors for any specific document typed by that operator, the frequency of errors averaged over the plurality of documents.
14. The quality management system of claim 13 wherein the means for assigning documents to different groups of operators comprises means for randomly assigning documents to different groups of operators.
15. The quality management system of claim 13 wherein each group of operators consists of a pair of operators.
16. The quality management system of claim 13 wherein the means for estimating an average frequency of errors for each operator comprises:
means estimating an average frequency of errors for each group of operators, based on the statistical analysis of documents assigned to that group of operators; and
means for, based on the assignment of operators to groups and the estimated average frequency of errors for each group, estimating the average frequency of errors for each operator.
17. The quality management system of claim 13 wherein:
the documents comprise fields; and
the means for estimating an average frequency of errors for each operator comprises means for estimating an average frequency of errors for at least one field for each operator.
18. The quality management system of claim 13 wherein the means for estimating an average frequency of errors comprises means for estimating an average frequency of errors for each field.