Enhanced beam search decoding for transformer-based OCR with probability score optimization

1 . A computer-implemented method for optical character recognition of imagery, the method comprising:

receiving an input image depicting an amount;

processing the input image through an encoder of a transformer-based optical character recognition model to generate encoded visual features;

performing sequential decoding of the amount using a decoder of the transformer-based optical character recognition model, wherein performing the sequential decoding comprises:

at each decoding step, generating, by the decoder, output values comprising raw logit scores for tokens in a vocabulary of possible tokens, the output values being conditioned on the encoded visual features and any previously decoded tokens;

converting the output values from raw logit scores to normalized confidence scores using a normalization function;

generating and maintaining a plurality of candidate output sequences based on the raw logit scores during the sequential decoding, each candidate output sequence comprising one or more decoded tokens, wherein maintaining the plurality of candidate output sequences comprises, at each decoding step, expanding candidate output sequences based on the raw logit scores, pruning lower-scoring candidate output sequences based on average logit scores to retain a predetermined number of highest-scoring candidate output sequences, and then reranking the retained candidate output sequences based on logit scores calculated by averaging logit scores;

for each candidate output sequence, calculating a cumulative confidence score by averaging the logit scores across all tokens of that candidate output sequence to enable reranking of the generated candidate output sequences;

selecting a final output sequence from among the plurality of candidate output sequences based on the cumulative confidence scores, the final output sequence having a maximum cumulative confidence score; and

outputting the final output sequence as a recognized textual representation of the amount.

2 . The method of claim 1 , wherein the final output sequence selected based on the maximum cumulative confidence score differs from a candidate output sequence that would have a highest cumulative logit score if cumulative logit scores were calculated by averaging the raw logit scores without conversion to normalized confidence scores.

3 . The method of claim 1 , wherein calculating the cumulative confidence score by averaging the logit scores comprises dividing a sum of the logit scores by a total number of tokens in the candidate output sequence.

4 . The method of claim 1 , wherein the expanding candidate output sequences based on the raw logit scores creates expanded candidate output sequences, the pruning is pruning of the expanded candidate output sequences, and maintaining the plurality of candidate output sequences further comprises, at each decoding step:

for each retained expanded candidate output sequence, calculating an updated cumulative confidence score by averaging logit scores across all tokens in that expanded candidate output sequence; and

wherein, the reranking the retained expanded candidate output sequences is based on the updated cumulative confidence scores.

5 . The method of claim 1 , wherein the predetermined number of highest-scoring expanded candidate output sequences is defined by a configurable beam width parameter.

6 . The method of claim 5 , wherein the configurable beam width parameter is set to a value of at least 3.

7 . The method of claim 5 , wherein the configurable beam width parameter is set to a value of 5.

8 . The method of claim 1 , wherein expanding each candidate output sequence comprises:

identifying a subset of tokens having highest logit score values from the raw logit scores generated at the decoding step; and

creating new candidate output sequences by appending each token in the subset to the candidate output sequence being expanded.

9 . The method of claim 1 , wherein processing the input image through the encoder comprises:

dividing the input image into a plurality of image patches;

flattening each image patch into a one-dimensional vector;

applying position embeddings to the flattened vectors to encode spatial position information; and

processing the flattened vectors with the position embeddings through a series of transformer encoder layers to generate the encoded visual features.

10 . The method of claim 1 , wherein the vocabulary of possible tokens includes a plurality of numerical digits, one or more special characters comprising any one or a combination of: a dollar sign, a dash, a slash, an asterisk, and a decimal point, and a control token comprising at least an end-of-sequence token.

11 . The method of claim 1 , wherein the method is performed on a user mobile device, and wherein the input image is captured by a camera of the user mobile device.

12 . The method of claim 1 , wherein the method is performed on a remote server, and wherein the input image is received from a user mobile device via a network connection.

13 . The method of claim 1 , further comprising:

comparing the recognized text representation of the amount to a courtesy amount or legal amount indicated in the input image; and

based on the comparison, determining whether to accept or reject a deposit transaction associated with the input image.

14 . The method of claim 1 , wherein the transformer-based optical character recognition model comprises:

a vision transformer encoder configured to process the input image and generate the encoded visual features; and

an autoregressive decoder configured to generate the output values at each decoding step based on the encoded visual features and previously decoded tokens.

15 . The method of claim 1 , wherein the normalized confidence scores at each decoding step sum to one across the entire vocabulary.

16 . The method of claim 1 , wherein maintaining the plurality of candidate output sequences comprises retaining a candidate output sequence that includes a lower normalized confidence score value at a first decoding step when one or more subsequent tokens in that candidate output sequence have higher normalized confidence scores that result in a higher cumulative confidence score for that candidate output sequence compared to other candidate output sequences.

17 . The method of claim 1 , wherein selecting the final output sequence based on cumulative confidence scores differs from a greedy decoding approach that selects a single highest-confidence token at each decoding step based on raw logit scores.

18 . The method of claim 1 , wherein the input image depicts the amount with one or more special characters selected from the group consisting of dollar signs, asterisks, and decimal points.

19 . A system for optical character recognition of imagery, the system comprising:

a processor; and

a non-transitory computer-readable memory storing instructions that, when executed by the processor, cause the system to:

receive an input image depicting an amount;

process the input image through an encoder of a transformer-based optical character recognition model to generate encoded visual features;

perform sequential decoding of the amount using a decoder of the transformer-based optical character recognition model, wherein performing the sequential decoding comprises:

at each decoding step, generating, by the decoder, output values comprising raw logit scores for tokens in a vocabulary of possible tokens, the output values being conditioned on the encoded visual features and any previously decoded tokens;

converting the output values from raw logit scores to normalized confidence scores using a normalization function;

generating and maintaining a plurality of candidate output sequences based on the raw logit scores during the sequential decoding, each candidate output sequence comprising one or more decoded tokens, and wherein maintaining the plurality of candidate output sequences comprises, at each decoding step, expanding candidate output sequences based on the raw logit scores, pruning lower-scoring candidate output sequences based on average logit scores to retain a predetermined number of highest-scoring candidate output sequences, and then reranking the retained candidate output sequences based on logit scores calculated by averaging logit scores;

for each candidate output sequence, calculating a cumulative confidence score by averaging the logit scores across all tokens of that candidate output sequence;

selecting a final output sequence from among the plurality of candidate output sequences based on the cumulative confidence scores, the final output sequence having a maximum cumulative confidence score; and

output the final output sequence as a recognized textual representation of the amount.

20 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

receiving an input image depicting an amount;

processing the input image through an encoder of a transformer-based optical character recognition model to generate encoded visual features;

performing beam search decoding of the amount using a decoder of the transformer-based optical character recognition model, wherein the beam search decoding comprises:

initializing a plurality of beams, each beam representing a candidate output sequence;

at each decoding step:

for each beam of the plurality of beams, generating, by the decoder, output values comprising raw logit scores for tokens in a vocabulary of possible next tokens, the output values being conditioned on the encoded visual features and tokens already present in that beam;

converting the output values from raw logit scores to normalized confidence scores using a normalization function;

expanding each beam based on the raw logit scores by appending possible next tokens selected according to the raw logit scores to create expanded candidate output sequences;

pruning the expanded candidate output sequences based on average logit scores to retain a predetermined beam width number of highest-scoring expanded candidate output sequences as beams for a subsequent decoding step;

for each retained expanded candidate output sequence, calculating a cumulative confidence score by averaging the logit scores across all tokens in that expanded candidate output sequence; and

reranking the retained expanded candidate output sequences based on the cumulative confidence scores;

continuing the beam search decoding until an end-of-sequence token is generated or a maximum sequence length is reached;

selecting a final output sequence having a maximum cumulative confidence score from among the beams; and

outputting the final output sequence as a recognized textual representation of the amount for use in remote processing.