Classifying handwritten math and text symbols using spatial syntactic rules, semantec connections, and ink-related information of strokes forming the symbols
The invention relates to a method implemented by a computing device for processing math and text in handwriting, comprising: identifying symbols by performing handwriting recognition on a plurality of strokes; classifying, as a first classification, first symbols as either a text symbol candidate or a math symbol candidate with a confidence score reaching a first threshold; classifying, as a second classification, second symbols other than first symbols as either a text symbol candidate or a math symbol candidate with a respective confidence score by applying predefined spatial syntactic rules; updating or confirming, as a third classification, a result of the second classification by establishing semantic connections between symbols and comparing the semantic connections with the result of the second classification; and recognising each symbol as either text or math based on a result of said third classification.
1. A method implemented by a computing device for processing math and text in handwriting, comprising:
identifying symbols by performing handwriting recognition on a plurality of strokes of digital ink;
classifying, as a first classification, at least one first symbol from the identified symbols as either a text symbol candidate or a math symbol candidate with a confidence score reaching at least a first threshold by applying individually predefined symbol recognition rules on the symbols;
classifying, as a second classification, at least one second symbol other than said at least one first symbol as either a text symbol candidate or a math symbol candidate with a respective confidence score by applying, on said at least one first symbol and at least one second symbol, predefined spatial syntactic rules defining spatial relationships between said at least one first symbol and said at least one second symbol;
updating or confirming, as a third classification, a result of said second classification by establishing semantic connections between at least two symbols from the identified symbols and comparing said semantic connections with the result of said second classification; and
recognising each symbol as either text symbol or math symbol based at least on a result of said third classification,
wherein said method comprises computing, for each identified symbol, ink-related information based on each stroke forming said symbols, said ink-related information comprising:
symbol information characterising intrinsic properties of the one or more strokes forming said symbols; and
context information characterising at least one neighbour stroke, spatially-related with the one or more strokes forming said symbol, in accordance with at least one respective ordering of the plurality of strokes;
wherein said ink-related information are used in at least said second and third classification to classify each symbol as either a math symbol candidate or a text symbol candidate.
2. The method of claim 1 comprising, prior to said identifying symbols, acquiring the plurality of strokes of digital ink forming the symbols,
said identified symbols comprising at least one text symbol and at least one math symbol.
3. The method of claim 1 , wherein during said identifying symbols, the plurality of strokes is compared with a set of predefined symbols and each symbol formed by the plurality of strokes is identified as a respective one within a set of predefined symbols.
4. The method of claim 1 , wherein said first classification comprises:
determining, for each identified symbol, embedding information attributed by the symbol recognition rules to said symbol, said embedding information comprising a respective confidence score that said predefined symbol is either text or math; and
comparing the respective confidence score of each identified symbol as defined in the embedding information with the first threshold to identify at least one first symbol as either a text symbol candidate or a math symbol candidate with a confidence score reaching at least said first threshold.
5. The method of claim 1 , during said second classification, said spatial syntactic rules are applied on said at least one first symbol to identify as either a math symbol candidate or a text symbol candidate at least one other symbol having a predefined spatial relationships with one or more first symbols.
6. The method of claim 1 , wherein during said second classification, each symbol other than said at least one first symbol is classified as either a text symbol candidate or a math symbol candidate.
7. The method of claim 1 , wherein the spatial syntactic rules comprise:
math spatial syntactic rules defining spatial relationships between at least two math symbols; and
text spatial syntactic rules defining spatial relationships between at least two text symbols.
8. The method of claim 1 , wherein if the result of the second classification is updated during the third classification based on semantic connections, the result produced in the third classification comprises a modified classification of the symbols as text and symbol candidates relative to the second classification along with respective updated confidence scores,
the method further comprising repeating the second classification on the result of the third classification to either confirm or update said result.
9. The method of claim 8 , wherein an iterative classifying process comprising the second and third classifications is repeated until said second and third classifications converge to a same result.
10. The method of claim 8 , wherein if a predefined number N of iterations of the classifying process is reached before the second and third classifications converge to a same result, the latest updated result of the second classification obtained in the third classification is used for recognising each symbol as either text symbol or math symbol, wherein N is an integer of at least 2.
11. The method of claim 1 , wherein said recognising each symbol as either text symbol or math symbol comprises allocating to each stroke, based at least on the result of said third classification, a classification as either math or text along with an associated confidence score.
12. The method of claim 1 , comprising grouping, based on a result of said recognising, the plurality of strokes into blocks of one or more strokes, each block being either a text block or a math block, wherein each stroke contained in a text block is a part of a text symbol and each stroke contained in a math block is part of a math symbol.
13. The method of claim 12 , comprising recognising math content and text content in said strokes by applying a math language model to each math block and by applying a text language model to each text block.
14. A computing device for processing math and text in handwriting, comprising:
a first recognition module for identifying symbols by performing handwriting recognition on a plurality of strokes of digital ink;
a symbol processing module for classifying, as a first classification, at least one first symbol from the identified symbols as either a text symbol candidate or a math symbol candidate with a confidence score reaching at least a first threshold by applying individually predefined symbol recognition rules on the symbols;
a syntax processing module for classifying, as a second classification, at least one second symbol other than said at least one first symbol as either a text symbol candidate or a math symbol candidate with a respective confidence score by applying, on said at least one first symbol and said at last one second symbol, predefined spatial syntactic rules defining spatial relationships between said at least one first symbol and said at least one second symbol; and
a semantic processing module for updating or confirming, as a third classification, a result of said second classification by establishing semantic connections between at least two symbols from the identified symbols and comparing said semantic connections with the result of said second classification;
said computing device being configured to recognise each symbol as either text symbol or math symbol based at least on a result of said third classification,
wherein said computing device is configured to compute, for each identified symbol, ink-related information based on each stroke forming said symbols, said ink-related information comprising:
symbol information characterising intrinsic properties of the one or more strokes forming said symbol; and
context information characterising at least one neighbour stroke, spatially-related with the one or more strokes forming said symbol, in accordance with at least one respective ordering of the plurality of strokes;
wherein said ink-related information are used in at least said second and third classification to classify each symbol as either a math symbol candidate or a text symbol candidate.