Natural language processing text-image-layout transformer
Disclosed herein is a system, method, and storage medium for Natural Language Processing (NLP) of real-world documents via a cloud data platform. The system combines three NLP models, including an encoder-decoder model, a spatial model, and a multi-modal model not previously combined. A text-image-layout transfer NLP system receives multi-modal input data and trains the multi-modal input data using the combination of the three NLP models.
1. A method comprising:
providing access to a machine learning model for iterative training on Natural Language Processing (NLP) of real-world documents, the providing access to the machine learning model comprising:
receiving, at a text-image-layout transformer (TILT) NLP system of a cloud data platform, multi-modal input data comprising text data, layout data, and image data;
executing multiple NLP models on the multi-modal input data, the multiple NLP models comprising:
an encoder-decoder model configured to generate text-based features not present in the text data;
a spatial model configured to implement spatial relationship features in the layout data; and
a multi-modal model configured to add visual context features to process the image data;
receiving additional data associated with the multi-modal input data, the additional data comprising semantic data associated with the text-based features, sequential distance data associated with the spatial relationship features, and spatial relationship data associated with the visual context features;
maintaining a distinction between the semantic data, the sequential distance data, and the spatial relationship data in the real-world documents, the maintaining the distinction comprising separating the semantic data from the sequential distance data and the spatial relationship data;
providing regularization augmentation to each of the text-based features, the spatial relationship features, and the visual context features while enabling cross-modal learning among the encoder-decoder model, the spatial model, and the multi-modal model; and
training the machine learning model on the multi-modal input data, the text-based features, the spatial relationship features, and the visual context features.
2. The method of claim 1 , further comprising:
processing the multi-modal input data, the processing comprising maintaining distinct processing paths for the text-based features extracted from the text data, the spatial relationship features derived from the layout data, and the visual context features extracted from the image data; and
unifying the distinct processing paths through an end-to-end neural architecture to preserve independence of the text-based features, the spatial relationship features, and the visual context features.
3. The method of claim 1 , wherein the training on the NLP of the real-world documents comprises:
analyzing the multi-modal input data; and
receiving at least one question regarding the multi-modal input data.
4. The method of claim 3 , further comprising:
generating output including at least one of answers to the at least one question, key information, and document classification.
5. The method of claim 1 , wherein the TILT NLP system of the cloud data platform performs operations comprising:
receiving the real-world documents;
extending biases with spatial relationships that include relative attention biases; and
providing additional image semantics to the received real-world documents.
6. The method of claim 1 , further comprising:
employing spatial bias augmentation, wherein biases are extended with spatial relationships; and
generating contextualized image embeddings, wherein additional image semantics are provided with the multi-modal input data.
7. The method of claim 6 , further comprising:
embedding distributional and contextualized semantics of a text token into a multi-dimensional vector space;
adding text embeddings to visual features of the multi-modal input data; and
assigning, to the text token, the visual features relative to a position and surrounding in the multi-dimensional vector space.
8. A system comprising:
one or more hardware processors of a machine; and
at least one memory storing instructions that, when executed by the one or more hardware processors, cause the machine to perform operation comprising:
providing access to a machine learning model for iterative training on Natural Language Processing (NLP) of real-world documents, the providing access to the machine learning model comprising:
receiving, at a text-image-layout transformer (TILT) NLP system of a cloud data platform, multi-modal input data comprising text data, layout data, and image data;
executing multiple NLP models on the multi-modal input data, the multiple NLP models comprising:
an encoder-decoder model configured to generate text-based features not present in the text data;
a spatial model configured to implement spatial relationship features in the layout data; and
a multi-modal model configured to add visual context features to process the image data;
receiving additional data associated with the multi-modal input data, the additional data comprising semantic data associated with the text-based features, sequential distance data associated with the spatial relationship features, and spatial relationship data associated with the visual context features;
maintaining a distinction between the semantic data, the sequential distance data, and the spatial relationship data in the real-world documents, the maintaining the distinction comprising separating the semantic data from the sequential distance data and the spatial relationship data;
providing regularization augmentation to each of the text-based features, the spatial relationship features, and the visual context features while enabling cross-modal learning among the encoder-decoder model, the spatial model, and the multi-modal model; and
training the machine learning model on the multi-modal input data, the text-based features, the spatial relationship features, and the visual context features.
9. The system of claim 8 , the operations further comprising:
processing the multi-modal input data, the processing comprising maintaining distinct processing paths for the text-based features extracted from the text data, the spatial relationship features derived from the layout data, and the visual context features extracted from the image data; and
unifying the distinct processing paths through an end-to-end neural architecture to preserve independence of the text-based features, the spatial relationship features, and the visual context features.
10. The system of claim 8 , wherein the training on the NLP of the real-world documents further comprises:
analyzing the multi-modal input data; and
receiving at least one question regarding the multi-modal input data.
11. The system of claim 10 , the operations further comprising:
generating output including at least one of answers to the at least one question, key information, and document classification.
12. The system of claim 8 , wherein the TILT NLP system of the cloud data platform performs operations further comprising:
receiving the real-world documents;
extending biases with spatial relationships that include relative attention biases; and
providing additional image semantics to the received real-world documents.
13. The system of claim 8 , the operations further comprising:
employing spatial bias augmentation, wherein biases are extended with spatial relationships; and
generating contextualized image embeddings, wherein additional image semantics are provided with the multi-modal input data.
14. The system of claim 13 , the operations further comprising:
embedding distributional and contextualized semantics of a text token into a multi-dimensional vector space;
adding text embeddings to visual features of the multi-modal input data; and
assigning, to the text token, the visual features relative to a position and surrounding in the multi-dimensional vector space.
15. A non-transitory computer readable storage medium embodying instructions that, when executed by a machine, cause the computer to perform operations comprising:
providing access to a machine learning model for iterative training on Natural Language Processing (NLP) of real-world documents, the providing access to the machine learning model comprising:
receiving, at a text-image-layout transformer (TILT) NLP system of a cloud data platform, multi-modal input data comprising text data, layout data, and image data;
executing multiple NLP models on the multi-modal input data, the multiple NLP models comprising:
an encoder-decoder model configured to generate text-based features not present in the text data;
a spatial model configured to implement spatial relationship features in the layout data; and
a multi-modal model configured to add visual context features to process the image data;
receiving additional data associated with the multi-modal input data, the additional data comprising semantic data associated with the text-based features, sequential distance data associated with the spatial relationship features, and spatial relationship data associated with the visual context features;
maintaining a distinction between the semantic data, the sequential distance data, and the spatial relationship data in the real-world documents, the maintaining the distinction comprising separating the semantic data from the sequential distance data and the spatial relationship data;
providing regularization augmentation to each of the text-based features, the spatial relationship features, and the visual context features while enabling cross-modal learning among the encoder-decoder model, the spatial model, and the multi-modal model; and
training the machine learning model on the multi-modal input data, the text-based features, the spatial relationship features, and the visual context features.
16. The non-transitory computer readable storage medium of claim 15 , the operations further comprising:
processing the multi-modal input data, the processing comprising maintaining distinct processing paths for the text-based features extracted from the text data, the spatial relationship features derived from the layout data, and the visual context features extracted from the image data; and
unifying the distinct processing paths through an end-to-end neural architecture to preserve independence of the text-based features, the spatial relationship features, and the visual context features.
17. The non-transitory computer readable storage medium of claim 15 , wherein the training on the NLP of the real-world documents comprises:
analyzing the multi-modal input data; and
receiving at least one question regarding the multi-modal input data.
18. The non-transitory computer readable storage medium of claim 17 , the operations further comprising:
generating output including at least one of answers to the at least one question, key information, and document classification.
19. The non-transitory computer readable storage medium of claim 15 , wherein the TILT NLP system of the cloud data platform performs operations further comprising:
receiving the real-world documents;
extending biases with spatial relationships that include relative attention biases; and
providing additional image semantics to the received real-world documents.
20. The non-transitory computer readable storage medium of claim 15 , the operations further comprising:
employing spatial bias augmentation, wherein biases are extended with spatial relationships; and
generating contextualized image embeddings, wherein additional image semantics are provided with the multi-modal input data.