SYSTEM AND METHOD FOR EXTRACTING CONTENT FROM RECORDS
The present disclosure is directed toward systems, methods, and non-transitory computer-readable media for generating bounding boxes for an image in accordance with one or more embodiments. The disclosed systems can generate a set of hidden states for an image of a document. The disclosed systems can generate a set of bounding boxes and a set of tokens from the hidden states utilizing a first head component and a second head component of a record generation model. The disclosed systems can aggregate a plurality of bounding boxes into a bounding box group. Additionally, the disclosed systems can provide the image depicting the bounding box group for display within a graphical user interface of a client device. Additionally or alternatively, the disclosed systems can train the record generation model, in part, by determining that a loss associated with masking one or more training bounding boxes satisfies a threshold loss.
1 . A computer-implemented method comprising:
generating, utilizing a transformer component and a decoder component of a record generation model, a set of hidden states for an image of a document from a client device;
generating, utilizing a first head component and a second head component of the record generation model, a set of bounding boxes and a set of tokens from the set of hidden states, each bounding box of the set of bounding boxes corresponding to a respective token of the set of tokens;
aggregating, utilizing a matcher component of the record generation model, a plurality of bounding boxes of the set of bounding boxes into a bounding box group that corresponds to a subset of tokens of the set of tokens; and
providing, for display within a graphical user interface of the client device, the image depicting the bounding box group that corresponds to the subset of tokens.
2 . The computer-implemented method of claim 1 , further comprising:
pairing, utilizing the matcher component, each bounding box of the bounding box group to a respective token of the subset of tokens based on the respective token corresponding to a field tag or a record tag.
3 . The computer-implemented method of claim 1 , wherein generating the set of bounding boxes and the set of tokens further comprises:
generating, from a subset of the set of hidden states, one or more bounding boxes of the set of bounding boxes; and
generating, from the subset of the set of hidden states, one or more tokens that correspond to the one or more bounding boxes.
4 . The computer-implemented method of claim 1 , wherein generating the set of bounding boxes and the set of tokens further comprises:
generating the set of tokens in accordance with a markup language format comprising content tokens indicating text extracted from the image of the document, record tags indicating that one or more content tokens associated with a record tag correspond to a record in the image of the document, and field tags indicating that one or more content tokens associated with a field tag correspond to a field in the image of the document.
5 . The computer-implemented method of claim 1 , further comprising:
generating a depiction of the bounding box group as a positional rectangle enclosing a portion of the image comprising information associated with a field or record of the image, wherein the subset of tokens of the set of tokens corresponds to the information associated with the field or record of the image.
6 . The computer-implemented method of claim 1 , further comprising:
generating the set of bounding boxes, the set of tokens, the set of hidden states, or a combination thereof utilizing the record generation model trained by:
generating a set of training tokens and a set of training boxes for a training image;
masking one or more training boxes in the set of training boxes; and
updating one or more parameters of the record generation model based on a loss associated with masking the one or more training boxes.
7 . The computer-implemented method of claim 6 , wherein the record generation model is further trained by:
generating a prediction of data based on masking the one or more training boxes;
determining the loss associated with masking the one or more training boxes by comparing the prediction of data with training data associated with the training image; and
determining that a masked training box of the one or more training boxes corresponds to a subset of training tokens based on the loss satisfying a threshold loss.
8 . A system comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:
generate, utilizing a transformer component and a decoder component of a record generation model, a set of hidden states for an image of a document from a client device;
generating, utilizing a first head component and a second head component of the record generation model, a set of bounding boxes and a set of tokens from the set of hidden states, each bounding box of the set of bounding boxes corresponding to a respective token of the set of tokens;
aggregating, utilizing a matcher component of the record generation model, a plurality of bounding boxes of the set of bounding boxes into a bounding box group that corresponds to a subset of tokens of the set of tokens; and
providing, for display within a graphical user interface of the client device, the image depicting the bounding box group that corresponds to the subset of tokens.
9 . The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
pair, utilizing the matcher component, each bounding box of the bounding box group to a respective token of the subset of tokens based on the respective token corresponding to a field tag or a record tag.
10 . The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to generate the set of bounding boxes and the set of tokens by:
generating, from a subset of the set of hidden states, one or more bounding boxes of the set of bounding boxes; and
generating, from the subset of the set of hidden states, one or more tokens that correspond to the one or more bounding boxes.
11 . The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to generate the set of bounding boxes and the set of tokens by:
generating the set of tokens in accordance with a markup language format comprising content tokens indicating text extracted from the image of the document, record tags indicating that one or more content tokens associated with a record tag correspond to a record in the image of the document, and field tags indicating that one or more content tokens associated with a field tag correspond to a field in the image of the document.
12 . The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generating a depiction of the bounding box group as a positional rectangle enclosing a portion of the image comprising information associated with a field or record of the image, wherein the subset of tokens of the set of tokens corresponds to the information associated with the field or record of the image.
13 . The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate the set of bounding boxes, the set of tokens, the set of hidden states, or a combination thereof utilizing the record generation model trained by:
generating a set of training tokens and a set of training boxes for a training image;
masking one or more training boxes in the set of training boxes; and
updating one or more parameters of the record generation model based on a loss associated with masking the one or more training boxes.
14 . The system of claim 13 , wherein the record generation model is further trained by:
generating a prediction of data based on masking the one or more training boxes;
determining the loss associated with masking the one or more training boxes by comparing the prediction of data with training data associated with the training image; and
determining that a masked training box of the one or more training boxes corresponds to a subset of training tokens based on the loss satisfying a threshold loss.
15 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to:
generate, utilizing a transformer component and a decoder component of a record generation model, a set of hidden states for an image of a document from a client device;
generate, utilizing a first head component and a second head component of the record generation model, a set of bounding boxes and a set of tokens from the set of hidden states, each bounding box of the set of bounding boxes corresponding to a respective token of the set of tokens;
aggregate, utilizing a matcher component of the record generation model, a plurality of bounding boxes of the set of bounding boxes into a bounding box group that corresponds to a subset of tokens of the set of tokens; and
provide, for display within a graphical user interface of the client device, the image depicting the bounding box group that corresponds to the subset of tokens.
16 . The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
pair, utilizing the matcher component, each bounding box of the bounding box group to a respective token of the subset of tokens based on the respective token corresponding to a field tag or a record tag.
17 . The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate, from a subset of the set of hidden states, one or more bounding boxes of the set of bounding boxes; and
generate, from the subset of the set of hidden states, one or more tokens that correspond to the one or more bounding boxes.
18 . The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate the set of tokens in accordance with a markup language format comprising content tokens indicating text extracted from the image of the document, record tags indicating that one or more content tokens associated with a record tag correspond to a record in the image of the document, and field tags indicating that one or more content tokens associated with a field tag correspond to a field in the image of the document.
19 . The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate a depiction of the bounding box group as a positional rectangle enclosing a portion of the image comprising information associated with a field or record of the image, wherein the subset of tokens of the set of tokens corresponds to the information associated with the field or record of the image.
20 . The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate the set of bounding boxes, the set of tokens, the set of hidden states, or a combination thereof utilizing the record generation model trained by:
generating a set of training tokens and a set of training boxes for a training image;
masking one or more training boxes in the set of training boxes; and
updating one or more parameters of the record generation model based on a loss associated with masking the one or more training boxes.