Handwriting recognition with language modeling
Systems and methods for handwriting recognition using language modeling facilitate improved results by using a trained language model to improve results from a handwriting recognition machine learning model. The language model may be a character-based language model trained on a dataset pertinent to field values on which the handwriting recognition model is to be used. A loss prediction module may be trained with the handwriting recognition model and/or the language model and used to determine whether a prediction from the handwriting recognition model should be refined by passing the prediction through the trained language model.
1. A hardware storage device having stored thereon computer-executable instructions that, when executed by one or more processors, cause a computing system to perform at least the following:
receive an image comprising handwriting;
generate, using a handwriting recognition model, a handwriting recognition prediction based on the image;
generate, using a loss prediction module, a loss prediction for the handwriting recognition prediction;
determine that the loss prediction does not exceed a predetermined threshold; and
generate, using a language model, an embedding of the handwriting recognition prediction.
2. The hardware storage device of claim 1 , further configured to perform at least the following:
generate, based on the embedding of the handwriting recognition prediction, an embedding-based prediction of a name depicted in the image.
3. The hardware storage device of claim 1 , further configured to perform at least the following:
search an embedding dictionary using the embedding of the handwriting recognition prediction to identify from the embedding dictionary a value related to the embedding.
4. The hardware storage device of claim 1 , wherein the language model is a character-based language model.
5. The hardware storage device of claim 1 , wherein the handwriting recognition model and the loss prediction module are trained together by performing at least the following:
during a training phase, receive an image comprising handwriting and a manual label corresponding to the image;
generate, using a handwriting recognition model, a handwriting recognition prediction based on the image;
determine a loss between the manual label and the handwriting recognition prediction and adjusting one or more components of the handwriting recognition model using the loss;
generate, using a loss prediction module, a loss prediction for the handwriting recognition prediction;
generate, using a language model, an embedding of the handwriting recognition prediction and of the corresponding manual label;
determine an embedding loss between the embeddings;
determine a loss-prediction loss between the embedding loss and the loss prediction; and
adjust one or more components of the loss prediction module using the loss-prediction loss.
6. The hardware storage device of claim 2 , wherein the language model is trained by performing at least the following:
receive a dataset;
transform the dataset with one or more changes; and
train the language model using a comparison between embeddings of corresponding fields in the dataset and the transformed dataset.
7. The hardware storage device of claim 6 , wherein the language model is chars2vec and the dataset is a names dataset.
8. The hardware storage device of claim 1 , wherein the loss prediction module is configured to receive a feature from a mid-level component of the handwriting recognition model.
9. The hardware storage device of claim 1 , wherein the loss prediction module is jointly learned with the handwriting recognition model.
10. A computer-implemented method for handwriting recognition comprising:
receiving an image comprising handwriting;
generating, using a handwriting recognition model, a handwriting recognition prediction based on the image;
generating, using a loss prediction module, a loss prediction for the handwriting recognition prediction;
determining that the loss prediction does not exceed a predetermined threshold;
generating, using a language model, an embedding based on the handwriting recognition prediction; and
searching an embedding dictionary using the embedding as a query to identify a related value.
11. The computer-implemented method for handwriting recognition of claim 10 , wherein the handwriting recognition model is a CNNRNNCTC-based model comprising a convolutional neural network (CNN) component, a recurrent neural network (RNN) component, and a connectionist temporal classification (CTC) component.
12. The computer-implemented method for handwriting recognition of claim 11 , wherein the loss prediction module receives a feature from one or more of the CNN component, the RNN component, and the CTC component.
13. The computer-implemented method for handwriting recognition of claim 11 , wherein the language model is a character-based language model trained based on a transformed dataset corresponding to the image comprising handwriting.
14. The computer-implemented method for handwriting recognition of claim 10 , wherein the related value is identified using an approximate nearest neighbor search.
15. The computer-implemented method for handwriting recognition of claim 13 , wherein the embedding dictionary is generated by passing a dataset corresponding to the image comprising handwriting through the trained language model.
16. A system comprising:
one or more processors; and
one or more hardware storage devices having stored thereon computer-executable instructions that, when executed by the one or more processors, configure the system to perform at least the following:
receive an image comprising handwriting;
generate, using a handwriting recognition model, a handwriting recognition prediction based on the image;
generate, using a loss prediction module, a loss prediction for the handwriting recognition prediction based on a feature from the handwriting recognition model;
determine that the loss prediction does not exceed a predetermined threshold;
generate, using a language model, an embedding based on the handwriting recognition prediction; and
search an embedding dictionary using the embedding as a query to identify a related value.
17. The system of claim 16 , wherein the handwriting recognition model is a CNNRNNCTC-based model comprising a CNN component, an RNN component, and a CTC component and the language model is a character-based language model trained based on a transformed dataset corresponding to the image comprising handwriting.
18. The system of claim 17 , wherein the loss prediction module receives the feature from one or more of the CNN, RNN, and CTC components of the handwriting recognition model.
19. The system of claim 16 , wherein the handwriting recognition model and the loss prediction module are jointly trained using (i) a comparison between the handwriting recognition prediction and a manual label corresponding to the handwriting recognition prediction and/or (ii) a comparison between the loss prediction and an embedding loss.
20. The system of claim 19 , wherein the embedding loss is determined by generating, using the language model, a handwriting recognition prediction embedding and a manual label embedding, and comparing the embeddings.