System and method for classifying connected groups of foreground pixels in scanned document images according to the type of marking

1. A method of classifying markings on an image of a document page according to marking type, the method comprising:

supplying the document page image to a segmenter;

breaking the image received by the segmenter into fragments of foreground pixel structures that are identified as being likely to be of the same marking type, by finding connected components, extracting near horizontal or vertical rule lines, and splitting at least some connected components to obtain the fragments;

supplying the fragments to a classifier, the classifier providing a category score to each fragment, wherein the classifier is trained from ground truth images whose pixels are labeled according to known marking types; and

assigning a same label to all pixels in a fragment when the fragment is classified by the classifier.

2. The method according to claim 1 wherein the segmenting step includes a first segmenting of horizontal and vertical lines which are detected and removed by morphological operations, and a second stage of recursively splitting the connected components until a size test is passed.

3. The method according to claim 2 wherein the segmenting is based on dynamic programming, to find a lowest cost split-path that traverses the fragment to be split.

4. The method according to claim 3 wherein the cost of the split-path is a sum of pixel costs along the path.

5. The method according to claim 1 wherein the fragments are smaller than a word.

6. The method according to claim 1 wherein the fragments are smaller than an individual letter or an individual number.

7. The method according to claim 1 , wherein preliminary category assignments are revised based on statistics of category assignments made to neighboring fragments.

8. The method according to claim 1 wherein parameters of the classifier are obtained by discriminative Adaboost training that selects useful features from among a family of measurements and assigns relative weights to the features.

9. The method according to claim 1 wherein the classifying detects printed text fragments, wherein one source of evidence is the degree of alignment observed between crests and bottoms of a fragment and those of the fragments neighbors.

10. A method of classifying markings on an image of a document page according to marking type, the method comprising:

supplying the document page image to a segmenter;

breaking the image received by the segmenter into fragments of foreground pixel structures that are identified as being likely to be of the same marking type, by finding connected components, extracting near horizontal or vertical rule lines, and splitting at least some connected components to obtain the fragments;

supplying the fragments to a classifier, the classifier providing a category score to each fragment, wherein the classifier is trained from ground truth images whose pixels are labeled according to known marking types; and

assigning a same label to all pixels in a fragment when the fragment is classified by the classifier, wherein the classifier is a two stage classifier, wherein in the first stage of the classifier each fragment is classified solely on the basis of features, resulting in each feature having a per category score, and the second stage classifies by taking into consideration secondary features.

11. The method according to claim 10 wherein the secondary features establish a relationship among category-labels of neighborhood fragments, while features used in the first stage of the classifier measure relationships among fragments and their observable properties.

12. A system of classifying markings on images in a document according to marking types, the system comprising:

a segmenter operated on a processor and configured to receive the document containing the images, the segmenter breaking the images into fragments of foreground pixel structures that are identified as being likely to be of the same marking type by finding connected components, and dividing at least some connected components to obtain image fragments; and

a classifier operated on a processor and configured to receive the fragments, the classifier providing a category score to each received fragment, wherein the classifier is trained from ground truth images whose pixels are labeled according to known marking types, the classifier assigning a same label to all pixels in a fragment when the fragment is classified by the classifier.

13. The system according to claim 12 further including a scanner to receive a hardcopy document containing images, the scanner converting the hardcopy document into an electronic document, the electronic document being the document supplied to the segmenter.

14. A system of classifying markings on images in a document according to marking types, the system comprising:

a segmenter operated on a processor and configured to receive the document containing the images, the segmenter breaking the images into fragments of foreground pixel structures that are identified as being likely to be of the same marking type by finding connected components, and dividing at least some connected components to obtain image fragments; and

a classifier operated on a processor and configured to receive the fragments, the classifier providing a category score to each received fragment, wherein the classifier is trained from ground truth images whose pixels are labeled according to known marking types, the classifier assigning a same label to all pixels in a fragment when the fragment is classified by the classifier, wherein the classifier is a two-stage classifier, and the first stage of the two-stage classifier operates on the fragment features and the second stage of the classifier is augmented by secondary features.

15. The system according to claim 14 wherein the first classifier stage consists of an array of one-vs-all classifiers, with one per category and produces an array of category scores between +1 and −1, and the second classifier stage is a score regularizer, which takes the category score array from the first classifier stage and produces an array of refined scores, informed by the scores of every marking type category.

16. A method of classifying markings on images in a document according to marking types, the method comprising:

supplying the document containing the images to a segmenter;

breaking the images received by the segmenter into fragments of foreground pixel structures that are identified as being likely to be of the same marking type by finding connected components, and dividing at least some connected components to obtain the fragments, and wherein the segmenting step includes a first segmenting of horizontal and vertical lines which are detected and removed by morphological operations, and a second stage of recursively splitting the connected components until a size test is passed;

supplying the fragments to a two stage classifier, the two stage classifier providing a category score to each fragment, wherein the two stage classifier is trained from ground truth images whose pixels are labeled according to known marking types, wherein the first classifier stage consists of an array of one-vs-all classifiers, with one per category and produces an array of category scores between +1 and −1, and the second classifier stage is a score regularizer, which takes the category score array from the first classifier stage and produces an array of refined scores, informed by the scores of every marking type category; and

assigning a same label to all pixels in a fragment when the fragment is classified by the classifier as a certain marking type.

17. The method according to claim 16 wherein the segmenting is based on dynamic programming, to find a lowest cost split-path that traverses the fragment to be split.

18. The method according to claim 17 wherein the cost of the split-path is a sum of pixel costs along the path.

19. The method according to claim 16 wherein the secondary features establish a relationship among category-labels of neighborhood fragments, while features used in the first stage of the classifier measure relationships among fragments and their observable properties.