Handwritten word spotter system using synthesized typed queries
View Patent ↗A wordspotting system and method are disclosed for processing candidate word images extracted from handwritten documents. In response to a user inputting a selected query string, such as a word to be searched in one or more of the handwritten documents, the system automatically generates at least one computer-generated image based on the query string in a selected font or fonts. A model is trained on the computer-generated image(s) and is thereafter used in the scoring the candidate handwritten word images. The candidate or candidates with the highest scores and/or documents containing them can be presented to the user, tagged, or otherwise processed differently from other candidate word images/documents.
1. A method comprising:
receiving a query string;
generating a plurality of computer-generated images based on the query string with a computer typographic font, each of the plurality of computer-generated images being generated by varying the computer typographic font, the font variations having been identified based on a precision for retrieving word images which match at least one selected query string;
training a model based on the plurality of computer-generated images;
scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and
based on the scores, identifying a subset of the word images.
2. The method of claim 1 , further comprising:
outputting the identified subset of word images.
3. The method of claim 1 , wherein the receiving of the query string is performed prior to training of the model based on the plurality of computer-generated images.
4. The method of claim 1 , wherein the model comprises a first model which is trained on the plurality of computer-generated images and a second model which has been trained on a collection of sample handwritten word images and wherein the score of each word image is a function of a computed likelihood of the word image with respect to the first model and a computed likelihood of the word image with respect to the second model.
5. The method of claim 1 , wherein the model comprises a semi-continuous hidden Markov model and optionally further comprises a Gaussian mixture model.
6. The method of claim 1 , wherein the generating of the plurality of computer-generated images based on the query string comprises generating binary images of a fixed pixel height in pixels and wherein the candidate handwritten word images are normalized to the same fixed pixel height and are binary images.
7. A non-transitory computer program product encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1 .
8. The method of claim 1 , further comprising:
extracting the collection of candidate handwritten word images from at least one document.
9. The method of claim 8 , wherein the extracting comprises scanning at least one handwritten document and segmenting the scanned handwritten document.
10. The method of claim 1 , wherein the training of the model based on the plurality of computer-generated images comprises extracting features from patches of the plurality of computer-generated images.
11. The method of claim 10 , wherein the extracting features from patches of the plurality of computer-generated images comprises translating the image stepwise with a window of fixed height and width and at each step, extracting a patch.
12. The method of claim 1 , wherein:
the generating of the plurality of computer-generated images based on the query string includes, for each of a set of fonts, automatically generating images based on the query string;
the training of the model based on the plurality of computer-generated images includes modeling the query string with a semi-continuous hidden Markov model, a subset of the parameters of the semi-continuous hidden Markov model being estimated based on features extracted from the images in the different fonts, and other parameters of the semi-continuous hidden Markov model being previously trained on sample handwritten word images without consideration of the query string;
the scoring of the candidate handwritten word images includes scoring candidate handwritten word images of the collection against the trained semi-continuous hidden Markov model; and
the identifying of the subset of the word images based on the scores includes labeling one or more of the candidate handwritten word images, or a document containing one or more of the candidate handwritten word images.
13. The method of claim 12 , wherein the subset of parameters includes transition probabilities and mixture weights and the other parameters include means and covariance matrices.
14. The method of claim 1 , wherein the collection of candidate handwritten word images comprises images extracted from at least one document.
15. The method of claim 14 , further comprising:
implementing a computer implemented process for at least one document from which a word image of the identified subset of the candidate handwritten word images was extracted.
16. The method of claim 15 , wherein the computer implemented process comprises displaying at least that portion of the document from which the word image was extracted.
17. The method of claim 15 , wherein the computer implemented process comprises processing a document which contains one of the subset of candidate word images differently from a document which does not.
18. A computer implemented processing system comprising:
a synthesizer which synthesizes a plurality of computer-generated images based on a received query string with a computer typographic font, each of the plurality of computer-generated images being generated by varying the computer typographic font, the font variations having been identified based on a precision for retrieving word images which match at least one selected query string;
a model which is trained on features extracted from the plurality of computer-generated images; and
a scoring component which scores candidate handwritten word images of a collection of candidate handwritten word images against the model and, based on the scores, identifies a subset of the handwritten word images.
19. The processing system of claim 18 , further comprising:
a document segmentor which processes an input document image to identify the candidate handwritten word images.
20. The processing system of claim 18 , further comprising:
a features extractor which extracts the features of the plurality of computer-generated images and the candidate handwritten word images.
21. The processing system of claim 18 , further comprising:
a document tagger which labels documents or portions thereof which contain at least one of the subset of handwritten word images.
22. The processing system of claim 18 , wherein the model comprises a semi-continuous hidden Markov model which includes a background model trained on sample handwritten word images prior to receipt of a query string.
23. A method comprising:
receiving a query string;
generating a plurality of computer-generated images based on the query string, each of the plurality of computer-generated images being generated by varying a selected computer typographic font;
training a model based on the plurality of computer-generated images in the different fonts;
scoring candidate handwritten word images of a collection of handwritten word images using the trained model; and
based on the scores, identifying a subset of the word images.
24. The method of claim 23 , further comprising identifying the set of fonts based on a precision of the set of fonts for retrieving word images which match at least one selected query string.