Granted Patent B2

US 12,619,828 · App. 18/563,002 · Granted May 5, 2026

Inventors: Lei Cui (Beijing, CN); Yiheng Xu (Beijing, CN); Yang Xu (Harbin, CN); Furu Wei (Beijing, CN); Zilong Wang (Shanghai, CN)

Assignee: Microsoft Technology Licensing, LLC

G06F40/30G06V30/414G06F40/106

Abstract

Claims (61)

1 . A computer-implemented method comprising:

executing a feature extraction neural network model that has been trained using example documents to determine a text sequence and layout information presented in a document, the text sequence comprising a plurality of text elements, the layout information indicating a spatial layout of the plurality of text elements in the document;

generating a plurality of semantic feature representations corresponding to the plurality of text elements based at least on the text sequence and the layout information; wherein using the trained feature extraction neural network model to generate the plurality of semantic feature representations comprises:

converting the text sequence and the layout information into a first embedding representation and a second embedding representation, respectively;

concatenating the first embedding representation and the second embedding representation, to obtain a concatenated embedding representation; and

applying the concatenated embedding representation into the trained feature extraction neural network model to generate the plurality of semantic feature representations; and

determining a reading order of the plurality of text elements in the document based on the plurality of semantic feature representations provided by the trained feature extraction neural network model.

2 . The method of claim 1 , wherein generating the plurality of semantic feature representations comprises:

for a first text element in the text sequence,

determining an attention weight for the first text element with respect to a second text element in the text sequence based on at least one of the following: a relative spatial positioning of the first text element with respect to the second text element in the document, and a relative ranking position of the first text element with respect to the second text element in the text sequence, the attention weight indicating an importance degree of the second text element to the first text element; and

determining a semantic feature representation of the first text element by weighting an embedding representation of the second text element with the determined attention weight.

3 . The method of claim 1 , wherein generating the plurality of semantic feature representations comprises:

determining an image-format file corresponding to the document;

determining visual information from the image-format file, the visual information indicating visual appearances of the plurality of text elements presented in the document; and

generating the plurality of semantic feature representations further based on the visual information.

4 . A computer-implemented method comprising:

executing a feature extraction neural network model that has been trained using example documents to determine a text sequence, layout information and order labeling information presented in a first sample document, the text sequence comprising a first set of text elements, the layout information indicating a spatial layout of the first set of text elements in the first sample document, the order labeling information indicating a ground-truth reading order of the first set of text elements in the first sample document;

generating, using the feature extraction neural network model, respective semantic feature representations of the first set of text elements based at least on the text sequence and the layout information;

executing an order determination neural network model to determine a predicted reading order of the first set of text elements in the first sample document based on the semantic feature representations; and

training the feature extraction neural network model and order determination neural network model based on a difference between the predicted reading order and the ground-truth reading order.

5 . The method of claim 4 , wherein the first sample document comprises an editable text document, and determining the order labeling information comprises:

determining format information corresponding to the editable text document, the format information at least indicating the ground-truth reading order of the first set of text elements.

6 . The method of claim 4 , wherein determining the layout information comprises:

determining a vector file corresponding to the first sample document; and

determining the layout information of the first set of text elements from the vector file.

7 . The method of claim 6 , wherein a plurality of text elements that occur at different positions in the first sample document and represent a same text are assigned with different indices, and the plurality of text elements are labeled with different colors in the vector file, the color with which each text element is labeled being determined based on the index assigned to the text element; and

wherein determining the layout information of the first set of text elements from the vector file comprises:

assigning, based on the indices and the colors assigned to the plurality of text elements, layout information determined from the vector file to the plurality of text elements extracted from the first sample document.

8 . The method of claim 4 , wherein generating the semantic feature representations comprises:

determining visual information from a first image-format file corresponding to the first sample document, the visual information representing visual appearances of the first set of text elements presented in the first sample document; and

generating, using the feature extraction neural network model, the semantic feature representations further based on the visual information.

9 . The method of claim 4 , further comprising obtaining the pre-trained feature extraction neural network model by:

determining a second image-format file corresponding to a second sample document, the second sample document comprising a second set of text elements;

generating, by masking at least one text element of the second set of text elements in the second image-format file, first masking information to indicate that the at least one text element is masked and other text elements of the second set of text elements are not masked;

determining, using the feature extraction neural network model, respective semantic feature representations of the second set of text elements;

determining second masking information based on the respective semantic feature representations of the second set of text elements, the second masking information indicating whether respective text elements of the second set of text elements are masked; and

pre-training the feature extraction model based on a difference between the first masking information and the second masking information.

10 . The method of claim 8 , wherein the feature extraction neural network model is further configured to generate a first visual feature representation of the first image-format file based on the text sequence, the layout information and the visual information,

wherein the method further comprises obtaining the pre-trained feature extraction neural network model by:

determining a third sample document, a third image-format file, and match labeling information, the match labeling information indicating whether the third image-format file matches with the third sample document;

generating, using the feature extraction neural network model, a second visual feature representation of the third image-format file based on the third sample document and the third image-format file;

determining, based on the second visual feature representation, a match result indicating whether the third image-format file matches with the third sample document; and

pre-training the feature extraction neural network model based on a difference between the match result and the match labeling information.

11 . An electronic device, comprising:

a processor, and

a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the device to perform acts comprising:

executing a feature extraction neural network model that has been trained using example documents to determine a text sequence and layout information presented in a document, the text sequence comprising a plurality of text elements, the layout information indicating a spatial layout of the plurality of text elements in the document;

generating a plurality of semantic feature representations corresponding to the plurality of text elements based at least on the text sequence and the layout information; wherein using the trained feature extraction neural network model to generate the plurality of semantic feature representations comprises:

for a first text element in the text sequence,

determining an attention weight for the first text element with respect to a second text element in the text sequence based on at least one of the following: a relative spatial positioning of the first text element with respect to the second text element in the document, and a relative ranking position of the first text element with respect to the second text element in the text sequence, the attention weight indicating to the trained feature extraction neural network model an importance degree of the second text element to the first text element; and

determining a semantic feature representation of the first text element by weighting an embedding representation of the second text element with the determined attention weight; and

determining a reading order of the plurality of text elements in the document based on the plurality of semantic feature representations provided by the trained feature extraction neural network model.

12 . An electronic device, comprising:

a processor; and

a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the device to perform acts comprising:

executing a feature extraction neural network model that has been trained using example documents to determine a text sequence, layout information and order labeling information presented in a first sample document, the text sequence comprising a first set of text elements, the layout information indicating a spatial layout of the first set of text elements in the first sample document, the order labeling information indicating a ground-truth reading order of the first set of text elements in the first sample document;

generating, using the feature extraction neural network model, respective semantic feature representations of the first set of text elements based at least on the text sequence and the layout information;

executing an order determination neural network model to determine a predicted reading order of the first set of text elements in the first sample document based on the semantic feature representations; and

training the feature extraction neural network model and order determination neural network model based on a difference between the predicted reading order and the ground-truth reading order.

13 . The device of claim 12 , wherein the first sample document comprises an editable text document, and determining the order labeling information comprises:

determining format information corresponding to the editable text document, the format information at least indicating the ground-truth reading order of the first set of text elements.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (56)