IP Library Granted Patent US 11,710,304
Granted Patent B2
US 11,710,304 · App. 17/893,963 · Granted Jul 25, 2023

Text recognition for a neural network

Inventor: Eitan Anzenberg (Emeryville, CA)
Assignee: BILL.COM, LLC
G06V10/82G06N3/08G06V10/454G06V30/19173G06V30/412G06V30/413G06V30/414
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,710,304
App. No.
17/893,963
Granted
Jul 25, 2023
Kind
B2
Abstract

Image data having text associated with a plurality of text-field types is received, the image data including target image data and context image data. The target image data including target text associated with a text-field type. The context image data providing a context for the target image data. A trained neural network that is constrained to a set of characters for the text-field type is applied to the image data. The trained neural network identifies the target text of the text-field type using a vector embedding that is based on learned patterns for recognizing the context provided by the context image data. One or more predicted characters are provided for the target text of the text-field type in response to identifying the target text using the trained neural network.

Claims (28)

1. One or more computer storage media having computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations of utilizing a trained neural network as a text recognition system, the operations comprising:

receiving, at the trained neural network, image data having text associated with a plurality of text-field types, the image data including target image data and context image data, the target image data including target text associated with a text-field type, the context image data providing a context for the target image data, the target image data including visual characteristics data, the visual characteristics data including one or more of: font size of the target text, color of the target text, and a format of the target text, wherein the context image data including one or more of: content surrounding the target text, font size of the content surrounding the target text, color of the content around the target text, a location of the target text relative to a location of the content, and spacing between the content and the target text;

applying, to the image data, the trained neural network, the trained neural network identifying the target text of the text-field type using a vector embedding that is based on learned patterns for recognizing the visual characteristics data and the context provided by the context image data; and

based at least in part on the learned patterns for recognizing the visual characteristics data and the context, providing one or more predicted characters for the target text of the text-field type in response to identifying the target text using the trained neural network.

2. The computer storage media of claim 1 , wherein the operations further comprising, determining that the target text is reoccurring in the image data, and wherein the providing one or more predicted characters for the target text of the text-field type is based on the determining that the target text is reoccurring.

3. The computer storage media of claim 1 , wherein the target text is an amount due.

4. The computer storage media of claim 1 , wherein the context image data includes a bounding box around the target text, shading around the target text, and a location of the target text in relationship to the location of other text.

5. The computer storage media of claim 1 , wherein the trained neural network includes a convolution neural network that is trained end-to-end with a recurrent neural network (RNN).

6. The computer storage media of claim 5 , wherein the trained neural network includes an interface that facilitates the RNN processing a particular number of text characters for the text-field type.

7. The computer storage media of claim 1 , wherein the image data includes a financial document of at least one of: a bill, invoice, receipt, or purchase order.

8. One or more computer storage media devices having computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations of training a neural network to identify relevant portions of text within an image, the operations comprising:

receiving training images comprising training image data having text, the training image data including target image data and context image data, the target image data including target text associated with a text-field type, the context image data providing a context for the target text, the target image data including visual characteristics data, the visual characteristics data including one or more of: font size of the target text, color of the target text, and a format of the target text, wherein the context image data including one or more of: content surrounding the target text, font size of the content surrounding the target text, color of the content around the target text, a location of the target text relative to a location of the content, and spacing between the content and the target text;

receiving an annotation for the training image data, the annotation visually indicating the target text captured by the target image data; and

training a neural network to learn patterns for recognizing the context provided by the context image data based at least in part on the training image data, the annotation for the training image data, and the visual characteristics data, wherein the trained neural network identifies new target text associated with the text-field type, the new target text identified based on using the learned patterns to generate a vector embedding of new image data, the new image data including new target image data and new context image data providing a new context for the target image data, and wherein the trained neural network provides one or more predicted characters for the new target text of the text-field type.

9. The one or more computer storage media devices of claim 8 , wherein the trained neural network identifies the target text of the text-field type based on the target text reoccurring in the image data.

10. The one or more computer storage media devices of claim 8 , wherein the trained neural network identifies the target text of the text-field type based on a relationship between text of the context image data and the target text.

11. The one or more computer storage media devices of claim 8 , wherein the context provided by the context image data includes at least one of an alpha-numeric character, a symbol, or a punctuation mark.

12. The one or more computer storage media devices of claim 8 , wherein the trained neural network includes a convolution neural network that is trained end-to-end with a recurrent neural network (RNN).

13. The one or more computer storage media devices of claim 12 , wherein the trained neural network includes an interface that facilitates the RNN processing a particular number of text characters for the text-field type.

14. The one or more computer storage media devices of claim 8 , wherein the one or more predicted characters includes at least two characters from the set of characters that constrain the neural network.

15. A computer-implemented method comprising:

receiving, at a trained neural network, an image comprising image data having text associated with a text-field type, the image data including target image data and context image data, the target image data including target text associated with the text-field type, the context image data providing a relationship between the target text and a remaining portion of the image data, the target image data including visual characteristics data, the visual characteristics data including one or more of: font size of the target text, color of the target text, and a format of the target text, wherein the context image data including one or more of: content surrounding the target text, font size of the content surrounding the target text, color of the content around the target text, a location of the target text relative to a location of the content, and spacing between the content and the target text; and

based at least in part on the visual characteristics data and the context image data, providing, via the trained neural network, one or more predicted characters for the target text of the text-field type in response to identifying the target text using the trained neural network.

16. The computer-implemented method of claim 15 , wherein the trained neural network identifies the target text of the text-field type based on the target text reoccurring in the image data.

17. The computer-implemented method of claim 15 , wherein the trained neural network includes a convolution neural network that is trained end-to-end with a recurrent neural network (RNN).

18. The computer-implemented method of claim 17 , wherein the trained neural network includes an interface that facilitates the RNN processing a particular number of text characters for the text-field type.

19. The computer-implemented method of claim 15 , wherein the trained neural network is a single channel neural network.

20. The computer-implemented method of claim 15 , wherein the one or more predicted characters includes at least two characters that constrain the trained neural network.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 27, 2023
From: BILL.COM, LLC
To: BILL OPERATIONS, LLC
Reel/Frame 065046/0884 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 23, 2022
From: ANZENBERG, EITAN
To: BILL.COM, LLC
Reel/Frame 060874/0893 →
Continuity (2)
Continuation 16882091 · May 22, 2020
Related Publication 20220406084A1 · Dec 22, 2022