Method and apparatus for recognizing text

1. A method for recognizing a text, the method comprising:

acquiring images of a text area of an input image, the acquired images including a text centerline graph, a text direction offset graph, a text boundary offset graph, and a text character classification graph;

extracting coordinates of feature points of a character center from the text centerline graph;

sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph to obtain a coordinate sequence of the feature points of the character center;

determining a polygonal bounding box of the text area based on the coordinate sequence of the feature points of the character center and the text boundary offset graph; and

determining a classification result of the feature points of the character center, based on the coordinate sequence of the feature points of the character center and the text character classification graph.

2. The method according to claim 1 , wherein the sorting comprises:

calculating a text reading direction based on the text direction offset graph; and

sorting the extracted coordinates of the feature points of the character center based on the text reading direction to obtain the coordinate sequence of the feature points of the character center.

3. The method according to claim 1 , wherein the method further comprises:

decoding the polygonal bounding box and the classification result of the feature points of the character center to obtain a recognition result of the text area.

4. The method according to claim 3 , wherein the method further comprises:

obtaining semantic information by analyzing the recognition result of the text area; and

adjusting the recognition result based on the semantic information to obtain the adjusted recognition result.

5. The method according to claim 4 , wherein the obtaining comprises:

performing point collection from the text character classification graph based on the coordinate sequence of the feature points of the character center, to obtain a semantic feature point of the feature points of the character center; and

constructing a semantic graph using the semantic feature points as a semantic node, and using a semantic adjacency relationship between the semantic feature points determined based on a distance between the semantic feature points as an edge connection.

6. The method according to claim 5 , wherein the adjusting comprises:

performing point collection from a convolution feature graph based on the coordinate sequence of the feature points of the character center, to obtain visual feature points of the feature points of the character center;

constructing a visual graph using the visual feature points as a visual node, and using a visual adjacency relationship between the visual feature points determined based on a distance between the visual feature points as an edge connection;

analyzing the semantic graph using a semantic graph network to obtain a semantic graph feature;

analyzing the visual graph using a visual graph network to obtain a visual graph feature;

fusing the semantic graph feature and the visual graph feature to obtain a fused feature; and

classifying the fused feature to obtain the adjusted recognition result.

7. The method according to claim 6 , wherein the semantic graph network and the visual graph network respectively use a connectionist temporal classification loss function for supervising and learning of training samples in a training phase.

8. The method according to claim 1 , wherein the acquiring comprises:

acquiring a convolution feature graph of the input image using a backbone convolutional network; and

extracting the text centerline graph, the text direction offset graph, the text boundary offset graph, and the text character classification graph from the convolution feature graph respectively using a multi-branch convolutional network.

9. The method according to claim 8 , wherein the multi-branch convolutional network comprises at least one of:

supervising and learning a first branch convolutional network based on a Dice coefficient loss function, wherein the convolution feature graph and the text centerline graph are an input to the first branch convolutional network and an output from the first branch convolutional network, respectively;

supervising and learning a second branch convolutional network based on a smooth L1 loss function, wherein the convolution feature graph and the text boundary offset graph are an input to the second branch convolutional network and an output from the second branch convolutional network, respectively;

supervising and learning a third branch convolutional network based on the smooth L1 loss function, wherein the convolution feature graph and the text direction offset graph are an input to the third branch convolutional network and an output from the third branch convolutional network, respectively; and

supervising and learning a fourth branch convolutional network based on a point collection-connectionist temporal classification loss function, wherein the convolution feature graph and the text character classification graph are an input to the fourth branch convolutional network and an output from the fourth branch convolutional network, respectively.

10. The method according to claim 1 , wherein:

extracting the coordinates comprises:

extracting the coordinates of the feature points of the character center from the text centerline graph using a pre-trained extraction module, or

the sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph comprises:

sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph to obtain the coordinate sequence of the feature points of the character center, using a pre-trained sorting module, or

the determining a polygonal bounding box comprises:

determining the polygonal bounding box of the text area based on the coordinate sequence of the feature points of the character center from the text boundary offset graph, using a bounding box picking module; or

the determining the classification result comprises:

determining the classification result of the feature points of the character center, based on the coordinate sequence of the feature points of the character center from the text character classification graph, using a character classification module.

11. An electronic device, comprising:

at least one processor; and

a memory, communicatively connected to the at least one processor; wherein,

the memory, storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, cause the at least one processor to perform an operation for recognizing a text, comprising:

acquiring images of a text area of an input image, the acquired images including a text centerline graph, a text direction offset graph, a text boundary offset graph, and a text character classification graph;

extracting coordinates of feature points of a character center from the text centerline graph;

sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph to obtain a coordinate sequence of the feature points of the character center;

determining a polygonal bounding box of the text area based on the coordinate sequence of the feature points of the character center and the text boundary offset graph; and

determining a classification result of the feature points of the character center, based on the coordinate sequence of the feature points of the character center and the text character classification graph.

12. A non-transitory computer readable storage medium, storing computer instructions, the computer instructions, being used to cause the computer to perform an operation for recognizing a text, comprising:

acquiring images of a text area of an input image, the acquired images including a text centerline graph, a text direction offset graph, a text boundary offset graph, and a text character classification graph;

extracting coordinates of feature points of a character center from the text centerline graph;

sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph to obtain a coordinate sequence of the feature points of the character center;

determining a polygonal bounding box of the text area based on the coordinate sequence of the feature points of the character center and the text boundary offset graph; and

determining a classification result of the feature points of the character center, based on the coordinate sequence of the feature points of the character center and the text character classification graph.

13. The medium according to claim 12 , wherein the sorting comprises:

calculating a text reading direction based on the text direction offset graph; and

sorting the extracted coordinates of the feature points of the character center based on the text reading direction to obtain the coordinate sequence of the feature points of the character center.

14. The medium according to claim 12 , wherein the operation further comprises:

decoding the polygonal bounding box and the classification result of the feature points of the character center to obtain a recognition result of the text area.

15. The medium according to claim 14 , wherein the operation further comprises:

obtaining semantic information by analyzing the recognition result of the text area; and

adjusting the recognition result based on the semantic information to obtain the adjusted recognition result.

16. The medium according to claim 15 , wherein the obtaining comprises:

performing point collection from the text character classification graph based on the coordinate sequence of the feature points of the character center, to obtain a semantic feature point of the feature points of the character center; and

constructing a semantic graph using the semantic feature points as a semantic node, and using a semantic adjacency relationship between the semantic feature points determined based on a distance between the semantic feature points as an edge connection.

17. The medium according to claim 16 , wherein the adjusting comprises:

performing point collection from a convolution feature graph based on the coordinate sequence of the feature points of the character center, to obtain visual feature points of the feature points of the character center;

constructing a visual graph using the visual feature points as a visual node, and using a visual adjacency relationship between the visual feature points determined based on a distance between the visual feature points as an edge connection;

analyzing the semantic graph using a semantic graph network to obtain a semantic graph feature;

analyzing the visual graph using a visual graph network to obtain a visual graph feature;

fusing the semantic graph feature and the visual graph feature to obtain a fused feature; and

classifying the fused feature to obtain the adjusted recognition result.

18. The medium according to claim 12 , wherein the acquiring comprises:

acquiring a convolution feature graph of the input image using a backbone convolutional network; and

extracting the text centerline graph, the text direction offset graph, the text boundary offset graph, and the text character classification graph from the convolution feature graph respectively using a multi-branch convolutional network.

19. The medium according to claim 18 , wherein the multi-branch convolutional network comprises at least one of:

supervising and learning a first branch convolutional network based on a Dice coefficient loss function, wherein the convolution feature graph and the text centerline graph are an input to the first branch convolutional network and an output from the first branch convolutional network, respectively;

supervising and learning a second branch convolutional network based on a smooth L1 loss function, wherein the convolution feature graph and the text boundary offset graph are an input to the second branch convolutional network and an output from the second branch convolutional network, respectively;

supervising and learning a third branch convolutional network based on the smooth L1 loss function, wherein the convolution feature graph and the text direction offset graph are an input to the third branch convolutional network and an output from the third branch convolutional network, respectively; and

supervising and learning a fourth branch convolutional network based on a point collection-connectionist temporal classification loss function, wherein the convolution feature graph and the text character classification graph are an input to the fourth branch convolutional network and an output from the fourth branch convolutional network, respectively.

20. The medium according to claim 12 , wherein:

extracting coordinates comprises:

extracting the coordinates of the feature points of the character center from the text centerline graph using a pre-trained extraction module, or

the sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph comprises:

sorting the extracted coordinates of the feature points of the character center based on the text direction offset graph to obtain the coordinate sequence of the feature points of the character center, using a pre-trained sorting module, or

the determining the polygonal bounding box comprises:

determining the polygonal bounding box of the text area based on the coordinate sequence of the feature points of the character center from the text boundary offset graph, using a bounding box picking module, or

the determining the classification result comprises:

determining the classification result of the feature points of the character center, based on the coordinate sequence of the feature points of the character center from the text character classification graph, using a character classification module.