Electronic document classification apparatus
View Patent ↗The apparatus computes classification scores based on parameters that have been determined from documents. Each score is compared with a first and second threshold. Definite classifications are assigned when the score is above the highest threshold or below the lowest threshold and the documents are processed accordingly. If the score is between the thresholds the document is singled out for further inspection, for example by a human arbitrator, to assign a class. The first and second threshold are adapted automatically based on specified a minimum accuracy level for the classification and a training set. The apparatus uses this specified accuracy in a search for a combination of threshold values that optimizes classifier yield, in terms of a maximized fraction of patterns in a training set that need not be turned over for further inspection without definite classification. The search is subject to the condition that the combination of thresholds results in at least the specified accuracy over the training set.
1. A document classification apparatus, comprising
a processor;
a score computation module configured to compute a score value from information obtained for a document to be classified;
a threshold comparator configured to perform a comparison to determine whether the score value is above a first threshold, below a second threshold or between the first and second threshold;
a further processing module configured to process the document dependent on the result of the comparison, the further processing module being configured to single out the document for further inspection to assign a class if the score value is between the first and second threshold;
a threshold determination module configured to determine the effective levels of the first and second threshold, based on an input minimum accuracy value and a set of training examples, by searching for a combination of effective thresholds levels that maximizes a yield over the training set while at least achieving said input minimum accuracy value, wherein the threshold determination module is configured to determine the yield according to
yield=( TP+TN+FP+FN )/( TP+TN+FP+FN+M )
and the accuracy according to a function of TP and TN and/or FP and FN that increases with increasing TP and TN and/or decreases with increasing FP and FN, wherein TP and FP are counts of training patterns with a score value above the first threshold that are identified in the training pattern to belongs to a class and not to belong to the class respectively, wherein FN and TN are counts of training patterns with a score value below the second threshold that are identified in the training pattern to belongs to the class and not to belong to the class respectively, and M is a count of training patterns with a score value between the first and second threshold.
2. A document classification apparatus as claimed in claim 1 , wherein the threshold determination module is configured to determine accuracy according to accuracy=(TP+TN)/(TP+TN+FP+FN).
3. A document classification apparatus, as claimed in claim 1 , wherein the score computation module is configured to compute a plurality of score values and the threshold comparator is configured to perform a plurality of determinations, each for a respective pairs of a first and second threshold, whether respective ones of the score values are above a first threshold, below a second threshold or between the first and second threshold of the pair, the threshold determination module configured to determine a plurality of the pairs of first and second effective threshold levels using said minimum accuracy for each of the determinations.
4. A computer implemented classification method, the method comprising
computing a score value from information obtained for an object to be classified;
determining whether the score value is above a first threshold, below a second threshold or between the first and second threshold;
applying further processing to the object dependent on the result of the comparison, singling out the object for further inspection to assign a class if the score value is between the first and second threshold;
determining effective levels of the first and second threshold based on an input minimum accuracy value and a set of training examples, by searching for a combination of effective threshold levels that maximizes a yield over the training set while at least achieving said input minimum accuracy value, the method comprising
counting template patterns to determine, for possible combinations of first and second effective threshold levels in the search, count values TP, FP, FN, TN and M, wherein TP and FP of training patterns with a score value above the first threshold that are identified in the training pattern to belongs to a class and not to belong to the class respectively, wherein FN and TN are counts of training patterns with a score value below the second threshold that are identified in the training pattern to belongs to the class and not to belong to the class respectively, and M is a count of training patterns with a score value between the first and second threshold, and
computing the yield in the search according to
yield=( TP+TN+FP+FN )/( TP+TN+FP+FN+M )
and the accuracy according to a function of TP and TN and/or FP and FN that increases with increasing TP and TN and/or decreases with increasing FP and FN.
5. A method as claimed in claim 4 , wherein the accuracy is determined according to accuracy=(TP+TN)/(TP+TN+FP+FN).
6. A non-transitory computer program product, comprising a program of instructions for a programmable computer including a processor, which when executed by the programmable computer cause the programmable computer to execute the method of claim 4 .
7. A non-transitory computer program product, comprising a program of instructions for a programmable computer including a processor, which when executed by the processor of the programmable computer cause the programmable computer to execute the method of claim 5 .
8. A document classification apparatus, as claimed in claim 2 , wherein the score computation module is configured to compute a plurality of score values and the threshold comparator is configured to perform a plurality of determinations, each for a respective pairs of a first and second threshold, whether respective ones of the score values are above a first threshold, below a second threshold or between the first and second threshold of the pair, the threshold determination module configured to determine a plurality of the pairs of first and second effective threshold levels using said minimum accuracy for each of the determinations.