System and method of character recognition using fully convolutional neural networks with attention
Embodiments of the present disclosure include a method that obtains a digital image. The method includes extracting a word block from the digital image. The method includes processing the word block by evaluating a value of the word block against a dictionary. The method includes outputting a prediction equal to a common word in the dictionary when a confidence factor is greater than a predetermined threshold. The method includes processing the word block and assigning a descriptor to the word block corresponding to a property of the word block. The method includes processing the word block using the descriptor to prioritize evaluation of the word block. The method includes concatenating a first output and a second output. The method includes predicting a value of the word block.
1. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing device, cause the computing device to:
receive a digital image corresponding to a word block, the word block containing at least one symbol corresponding to a letter, number, character, or a combination thereof;
evaluate the word block with a first neural network, the first neural network assigning at least one of a descriptor or a steering factor to the word block as a first output;
process the word block with a second neural network, the second neural network evaluating the steering factor to prioritize evaluation;
combine the first output from the first neural network and a second output from the second neural network;
process an input with a third neural network, the output of which is combined with the first and second outputs;
wherein the input to the third neural network comprises an entire text block, an entire line of text, an entire paragraph, an entire entry field of a form, an entire document, or an entire batch of documents, and;
output a predicted word corresponding to a value of the word block.
2. The non-transitory computer readable storage medium of claim 1 , wherein the instructions, when executed, further cause the computing device to:
resize the word block, the resized word block based on at least one of a height or a width of the word block;
predict a number of symbols and blanks in the word block; and
perform probabilistic correction on the word block.
3. The non-transitory computer readable storage medium of claim 1 , wherein the at least one of the first neural network or the second neural network comprise a convolutional neural network, a fully connected neural network, a recurrent neural network, a fully convolutional neural network, or a combination thereof.
4. The non-transitory computer readable storage medium of claim 1 , wherein the steering factor is a weight factor comprising at least one of attention weights, attention vectors, and Gaussian distributions.
5. The non-transitory computer readable storage medium of claim 1 , wherein the descriptor comprises at least one of handwriting style, slant, skew, rotation, blur, marker type, paper type, image fidelity, background threshold, or a combination thereof.
6. The non-transitory computer readable storage medium of claim 1 , wherein the first and second neural networks are convolutional neural networks.
7. The non-transitory computer readable storage medium of claim 1 , wherein the first and second neural networks are fully convolutional neural networks.
8. The non-transitory computer readable storage medium of claim 1 , wherein the first output from the first neural network and the second output from the second neural network are combined by concatenation.
9. The non-transitory computer readable storage medium of claim 1 , wherein the word output prediction is performed by a SoftMax layer.
10. The non-transitory computer readable storage medium of claim 1 , wherein the first neural network computes attention vectors of the word block.
11. The non-transitory computer readable storage medium of claim 10 , wherein the attention vectors are evaluated by the second neural network.
12. The non-transitory computer readable storage medium of claim 1 , wherein the word block comprises handwritten characters.
13. The non-transitory computer readable storage medium of claim 12 , wherein the second neural network is a fully convolutional handwritten character neural network.
14. The non-transitory computer readable storage medium of claim 1 , wherein the at least one descriptor is provided as a vector.