Granted Patent B2

US 12,633,147 · App. 18/567,243 · Granted May 19, 2026

Inventors: Dacheng Tao (Beijing, CN); Yue He (Beijing, CN)

Assignee: Jingdong Technology Information Technology Co., Ltd.

G06V30/19127G06F40/40G06V10/774G06V10/82G06V30/153G06V30/1918

Abstract

Claims (61)

1 . A method for recognizing a text, the method comprising:

acquiring feature maps, the feature maps being obtained by means of performing text instance segmentation on an image of a to-be-recognized text;

constructing a relationship graph according to the feature maps, wherein in the relationship graph: each node represents a pixel in a feature map, each edge represents that a similarity of spatial semantic features of two nodes connected by the edge is greater than a target threshold, and a spatial semantic feature of a node comprises a position feature and a type feature of a pixel represented by the node;

processing the relationship graph using a pre-trained graph convolutional network, to obtain a first text feature corresponding to the image; and

generating a text recognition result of the image based on the first text feature.

2 . The method according to claim 1 , wherein the method further comprises:

processing the feature maps using a pre-trained language model, to obtain a second text feature corresponding to the image; and

the generating a text recognition result of the image based on the first text feature, comprises:

generating the text recognition result of the image, based on the first text feature and the second text feature.

3 . The method according to claim 2 , wherein the generating the text recognition result of the image, based on the first text feature and the second text feature, comprises:

generating the text recognition result of the image based on the first text feature, the second text feature, and the feature maps.

4 . The method according to claim 3 , wherein the feature maps are at least two feature maps; and

the constructing a relationship graph according to the feature maps, comprises:

constructing, for each feature map in the at least two feature maps, a relationship subgraph corresponding to the feature map; and

merging relationship subgraphs corresponding to the feature maps to obtain the relationship graph.

5 . The method according to claim 4 , wherein a feature map in the at least two feature maps is used to represent an image feature of an image area in which a word of the to-be-recognized text is located and a sequential feature between the feature map and other feature maps.

6 . The method according to claim 5 , wherein each feature map is generated through following steps:

inputting the image to a convolutional neural network trained in advance to obtain an initial feature map;

performing text instance segmentation on the initial feature map to obtain instance feature maps;

performing text sequence segmentation on the initial feature map to obtain sequential feature maps; and

fusing the instance feature maps and the sequential feature maps to obtain the feature maps obtained by means of performing text instance segmentation on the image of the to-be-recognized text.

7 . The method according to claim 6 , wherein the convolutional neural network is implemented based on a feature pyramid network and a residual network, wherein a step size of at least one convolutional layer preceding to an output layer of the residual network is 1, and a feature map output by the residual network is subject to deformable convolution to generate an input feature map of the feature pyramid network.

8 . The method according to claim 2 , wherein the processing the feature maps using a pre-trained language model to obtain a second text feature corresponding to the image, comprises:

processing the feature maps using a semantic feature extraction network trained in advance to obtain a semantic feature corresponding to the feature maps, wherein the semantic feature extraction network comprises a pooling layer that performs pooling processing on the feature maps and a linear layer that performs linear transformation on an output of the pooling layer; and

inputting the semantic feature to the language model to obtain the second text feature.

9 . The method according to claim 3 , wherein the generating the text recognition result of the image, based on the first text feature, the second text feature, and the feature maps, comprises:

inputting the first text feature, the second text feature, and the feature maps to a pre-trained feature fusion network to generate the text recognition result of the image, wherein the feature fusion network is used for splicing the first text feature, the second text feature, and the feature maps, and performing linear transformation on a result obtained by the splicing to obtain the text recognition result of the image.

10 . The method according to claim 4 , wherein the spatial semantic feature of the node further comprises a sequential feature of a feature map in which the pixel represented by the node is located; and

for a node in a relationship subgraph, a spatial semantic feature of the node is generated through following operations:

acquiring a sequence value of a feature map corresponding to the node in the at least two feature maps;

determining, based on the sequence value, the sequential feature of the feature map corresponding to the node; and

splicing the position feature, the type feature, and the sequential feature of the pixel represented by the node to obtain the spatial semantic feature of the node.

11 . The method according to claim 4 , wherein a similarity of spatial semantic features of two nodes is determined through following operations:

determining an Euclidean distance between the two nodes, and determining a first similarity based on the Euclidean distance;

determining a cosine similarity of the type features corresponding to the two nodes; and

determining the similarity of spatial semantic features of the two nodes, based on the first similarity and the cosine similarity.

12 . The method according to claim 4 , wherein the merging relationship subgraphs corresponding to the feature maps, comprises:

connecting root nodes of neighboring relationship subgraphs in sequence according to a sequential relationship between the relationship subgraphs.

13 . The method according to claim 12 , wherein a root node of a feature map is determined through following operations:

determining, for a node in the feature map, an intersection and concatenation ratio between the node and each of other nodes, respectively, wherein the intersection and concatenation ratio represents a ratio of a number of elements included in an intersection of neighboring nodes of two nodes to a number of elements included in a concatenation of the neighboring nodes of the two nodes; and

in response to determining that respective intersection and concatenation ratios corresponding to the node are not greater than a preset threshold, determining the node as the root node.

14 . The method according to claim 4 , wherein the graph convolutional network comprises a first graph convolutional network and an output network, wherein the first graph convolutional network is used for transforming a feature matrix of the relationship graph, and the output network is used for selecting nodes from the respective relationship subgraphs based on an output of the first graph convolutional network, and aggregating transformed features corresponding to the nodes selected from the respective relationship subgraphs to obtain the first text feature.

15 . The method according to claim 14 , wherein the first graph convolutional network comprises a first graph convolutional subnetwork and a second graph convolutional subnetwork, wherein the first graph convolutional subnetwork is used for transforming the feature matrix of the relationship graph, and the second graph convolutional subnetwork is used for transforming, according to a relationship graph constructed based on an output of the first graph convolutional subnetwork, a feature matrix output by the first graph convolutional subnetwork.

16 . The method according to claim 6 , wherein the performing text instance segmentation on the initial feature map to obtain instance feature maps, comprises: performing text instance segmentation on the initial feature map using a pre-trained text instance segmentation network to obtain the instance feature maps; and

the performing text sequence segmentation on the initial feature map to obtain sequential feature maps, comprises: performing text sequence segmentation on the initial feature map using a pre-trained text sequence segmentation network to obtain the sequential feature maps; and

the graph convolutional network, the language model, and the segmentation network are obtained by training through following operations, wherein the segmentation network comprises the convolutional neural network, the text instance segmentation network and the text sequence segmentation network:

acquiring a labeled training dataset and an unlabeled training dataset, wherein training data in the labeled training dataset comprises labels of different granularities; and

constructing a teacher-student network corresponding to the graph convolutional network, the language model, and the segmentation network, and training the teacher-student network using the labeled training dataset, the unlabeled training dataset, and a preset loss function, wherein an input to the student network comprises labeled training data and unlabeled training data, and an input to the teacher network comprises the unlabeled training data, the loss function comprises a first loss function, a second loss function, and a third loss function, the first loss function and the second loss function respectively represent differences between output results of the student network for the labeled training data and the labels of different granularities, and the third loss function represents differences between output results of the student network and output results of the teacher network for the unlabeled training data, respectively.

17 . An apparatus for recognizing a text, the apparatus comprising:

one or more processors; and

a storage apparatus, storing one or more programs thereon,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement operations, the operations comprising:

acquiring feature maps, the feature maps being obtained by means of performing text instance segmentation on an image of a to-be-recognized text;

constructing a relationship graph according to the feature maps, wherein in the relationship graph: each node represents a pixel in a feature map, each edge represents that a similarity of spatial semantic features of two nodes connected by the edge is greater than a target threshold, and a spatial semantic feature of a node comprises a position feature and a type feature of a pixel represented by the node;

processing the relationship graph using a pre-trained graph convolutional network, to obtain a first text feature corresponding to the image; and

generating a text recognition result of the image based on the first text feature.

18 . A non-transitory computer readable medium, storing a computer program thereon, wherein, the program, when executed by a processor, implements operations, the operations comprising:

acquiring feature maps, the feature maps being obtained by means of performing text instance segmentation on an image of a to-be-recognized text;

constructing a relationship graph according to the feature maps, wherein in the relationship graph: each node represents a pixel in a feature map, each edge represents that a similarity of spatial semantic features of two nodes connected by the edge is greater than a target threshold, and a spatial semantic feature of a node comprises a position feature and a type feature of a pixel represented by the node;

processing the relationship graph using a pre-trained graph convolutional network, to obtain a first text feature corresponding to the image; and

generating a text recognition result of the image based on the first text feature.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (26)