Method and device for processing a multi-language text
Embodiments of the present disclosure provide a method and apparatus for processing a multi-language text. According to embodiments of the present disclosure, the multi-language text including contents in a plurality of languages may be encoded with a Unicode. The method further comprises splitting the multi-language text into a plurality of parts based on the Unicode of the multi-language text, contents of the plurality of parts having different languages. In addition, the multi-language text may also be processed based on the plurality of parts.
1. A method of processing a multi-language text, comprising:
encoding the multi-language text in a document with a Unicode, the multi-language text including contents in a plurality of languages;
splitting the multi-language text in the document into a plurality of parts by the plurality of languages used in the contents based on the Unicode of the multi-language text, contents of the plurality of parts having different languages, wherein splitting the multi-language text in the document into the plurality of parts comprises buffering, via a text part buffer, information regarding the Unicode and a position of a text part, of the plurality of parts, in at least a portion of original text, wherein splitting the multi-language text in the document into the plurality of parts comprises combining a plurality of code values, wherein the combined plurality of code values correspond to a language category; and
processing the multi-language text based on the plurality of parts.
2. The method of claim 1 , wherein splitting the multi-language text into a plurality of parts comprises:
obtaining a code value of the Unicode of the multi-language text, the code value corresponding to a content in the coded multi-language text; and
determining, based on the code value, a part of the plurality of parts to which the corresponding content belongs.
3. The method of claim 2 , wherein determining, based on the code value, a part of the plurality of parts to which the corresponding content belongs comprises:
matching the code value with one of a plurality of code value ranges in a reference table, the plurality of code value ranges corresponding to a plurality of languages; and
determining, based on the matching, the part to which the content corresponding to the code value belongs.
4. The method of claim 3 , wherein determining, based on the matching, the part to which the content corresponding to the code value belongs comprises:
in response to determining that the code value and a previous code value belong to different code value ranges, splitting the content corresponding to the code value into a new part of the plurality of parts; and
in response to determining that the code value and the previous code value belong to a same code value range, splitting the content corresponding to the code value into a same part of the plurality of parts as a previous content corresponding to the previous code value.
5. The method of claim 3 , wherein determining, based on the matching, the part to which the content corresponding to the code value belongs comprises: in response to determining that the code value belongs to a code value range corresponding to a common character, splitting the content corresponding to the code value into a same part of the plurality of parts as a previous content corresponding to a previous code value.
6. The method of claim 5 , wherein the common character includes at least one of a punctuation or a numeric value.
7. The method of claim 1 , wherein the multi-language text further includes a metadata, the method further comprising:
receiving an input to exclude the metadata from the splitting; and
in response to receiving the input, splitting only the contents of the multi-language text into the plurality of parts.
8. The method of claim 1 , wherein processing the multi-language text based on the plurality of parts comprises:
detecting a language of at least one part of the plurality of parts; and
processing the at least one part based on the detected language.
9. The method of claim 8 , wherein the at least one part includes two or more parts, the method further comprising: merging the two or more parts that are processed.
10. An electronic device, comprising:
at least one processing unit;
at least one memory coupled to the at least one processing unit and storing instructions executable by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to:
encode a multi-language text in a document with a Unicode, the multi-language text including contents in a plurality of languages;
split the multi-language text in the document into a plurality of parts by the plurality of languages used in the contents based on the Unicode of the multi-language text, contents of the plurality of parts having different languages, wherein splitting the multi-language text in the document into the plurality of parts comprises buffering, via a text part buffer, information regarding the Unicode and a position of a text part, of the plurality of parts, in at least a portion of original text, wherein splitting the multi-language text in the document into the wherein splitting the multi-language text in the document into the plurality of parts comprises combining a plurality of code values, wherein the combined plurality of code values correspond to a language category; and
process the multi-language text based on the plurality of parts.
11. The device of claim 10 , wherein splitting the multi-language text into a plurality of parts comprises:
obtaining a code value of the Unicode of the multi-language text, the code value corresponding to a content in the coded multi-language text; and
determining, based on the code value, a part of the plurality of parts to which the corresponding content belongs.
12. The device of claim 11 , wherein determining, based on the code value, a part of the plurality of parts to which the corresponding content belongs comprises:
matching the code value with one of a plurality of code value ranges in a reference table, the plurality of code value ranges corresponding to a plurality of languages; and
determining, based on the matching, the part to which the content corresponding to the code value belongs.
13. The device of claim 12 , wherein determining, based on the matching, the part to which the content corresponding to the code value belongs comprises:
in response to determining that the code value and a previous code value belong to different code value ranges, splitting the content corresponding to the code value into a new part of the plurality of parts; and
in response to determining that the code value and the previous code value belong to a same code value range, splitting the content corresponding to the code value into a same part of the plurality of parts as a previous content corresponding to the previous code value.
14. The device of claim 12 , wherein determining, based on the matching, the part to which the content corresponding to the code value belongs comprises: in response to determining that the code value belongs to a code value range corresponding to a common character, splitting the content corresponding to the code value into a same part of the plurality of parts as a previous content corresponding to a previous code value.
15. The device of claim 14 , wherein the common character includes at least one of a punctuation or a numeric value.
16. The device of claim 10 , wherein the multi-language text further includes a metadata, and the instructions, when executed by the at least one processing unit, further cause the device to:
receive an input to exclude the metadata from the splitting; and
in response to receiving the input, split only the contents of the multi-language text into the plurality of parts.
17. The device of claim 10 , wherein processing the multi-language text based on the plurality of parts comprises:
detecting a language of at least one part of the plurality of parts; and
processing the at least one part based on the detected language.
18. The device of claim 17 , wherein the at least one part includes two or more parts, and the instructions, when executed by the at least one processing unit, further cause the device to: merge the two or more parts that are processed.
19. A computer program product for processing a multi-language text, the computer program product comprising:
a non-transitory computer readable medium encoded with computer-executable program code for using read signatures in replication, the code configured to enable the execution of:
encoding the multi-language text in a document with a Unicode, the multi-language text including contents in a plurality of languages;
splitting the multi-language text in the document into a plurality of parts by the plurality of languages used in the contents based on the Unicode of the multi-language text, contents of the plurality of parts having different languages, wherein splitting the multi-language text in the document into the plurality of parts comprises buffering, via a text part buffer, information regarding the Unicode and a position of a text part, of the plurality of parts, in at least a portion of original text, wherein splitting the multi-language text in the document into the wherein splitting the multi-language text in the document into the plurality of parts comprises combining a plurality of code values, wherein the combined plurality of code values correspond to a language category; and
processing the multi-language text based on the plurality of parts.
20. The method of claim 9 wherein merging the two or more parts that are processed includes merging each processed part of the two or more parts into a single stream as a processing result of an entirety of the multi-language text.