Text processing method, device and storage medium
Provided are a text processing method, a device and a storage medium, relating to a field of computer technology, and especially to a field of artificial intelligence, such as natural language processing and deep learning. The specific implementation scheme includes: performing text processing on first text, by using a text processing acceleration operator; and processing, in parallel and faster, content after the text processing, by using the text processing acceleration operator. Text processing and parallel acceleration are carried out by the text processing acceleration operator, which can improve the speed of text processing.
1. A text processing method, comprising:
performing text processing on first text, by using a text processing acceleration operator, comprising:
obtaining a first tensor corresponding to the first text;
segmenting the first tensor, to obtain a plurality of segmentation results;
performing identifier mapping on each of the plurality of segmentation results; and
generating a second tensor, by using a mapping result; and
performing parallel acceleration processing on, content after the text processing, by using the text processing acceleration operator, comprising:
processing, in parallel, the second tensor, by using multiple threads.
2. The method of claim 1 , wherein segmenting the first tensor, to obtain the plurality of segmentation results, comprises at least one of:
performing word segmentation processing on the first tensor, to obtain a plurality of segmented words; and
performing term segmentation processing on the first tensor, to obtain a plurality of segmented terms.
3. The method of claim 2 , wherein performing the identifier mapping on each of the plurality of segmentation results, comprises:
mapping each segmented word and/or each segmented term to a numeric value, according to a mapping relationship.
4. The method of claim 1 , wherein generating the second tensor by using the mapping result, comprises:
obtaining each line of numeric value of the second tensor, according to a result of mapping of a corresponding line of string of the first tensor, the first tensor comprising N lines of string, the second tensor comprising N lines of numeric value corresponding to the N lines of string, and N being a positive integer.
5. The method of claim 1 , wherein the first text comprises a string, and the first tensor is a string tensor; and the second tensor is an integer tensor, and each line of numeric value of the second tensor uniquely corresponds to one line of string of the first tensor.
6. The method of claim 1 , wherein the text processing is transformed from being executed in a first language into being executed in a second language.
7. The method of claim 6 , wherein the first language is an interpreted language and the second language is a compiled language.
8. The method of claim 1 , further comprising:
performing natural language understanding processing on content after the parallel acceleration processing by using a natural language understanding model.
9. The method of claim 8 , wherein the natural language understanding model comprises the text processing acceleration operator and a transformer encoder operator.
10. The method of claim 9 , wherein the transformer encoder operator comprises a Fused Attention operator and a Fused Feed forward operator.
11. The method of claim 1 , further comprising:
performing natural language generating processing on content processed by the natural language understanding model, by using a natural language generating model.
12. The method of claim 11 , wherein the natural language generating model comprises a transformer decoder operator and a decoding strategy operator.
13. The method of claim 12 , wherein performing the natural language generating processing on the content processed by the natural language understanding model, by using the natural language generating model, comprises:
identifying a characteristic of a decoder code, by using the transformer decoder operator, to identify an adopted generation network type; and
calling a preset decoding operator corresponding to the generation network type.
14. The method of claim 13 , wherein the generation network type comprises at least one of:
a Decoder type;
a Prefix Language Model (Prefix LM) type; or
an Encoder-Decoder type.
15. The method of claim 12 , wherein performing the natural language generating processing on the content processed by the natural language understanding model, by using the natural language generating model, further comprises:
identifying a characteristic of a decoding strategy, by using the decoding strategy operator, to identify an adopted decoding algorithm; and
automatically translating a cyclic code running in a first language into a code of a second language, by using the decoding algorithm.
16. The method of claim 15 , wherein the decoding algorithm comprises at least one of:
a search-based decoding algorithm; or
a sampling-based decoding algorithm.
17. The method of claim 15 , wherein performing the natural language generating processing on the content processed by the natural language understanding model, by using the natural language generating model, further comprises:
compiling the code of the second language, by calling a Just In-time Compiler ability of a deep learning framework, to obtain a dynamic link library, so as to link the dynamic link library with the deep learning framework.
18. The method of claim 15 , wherein the first language is an interpreted language and the second language is a compiled language.
19. The method of claim 1 , further comprising:
exporting a word-list of the text processing acceleration operator.
20. The method of claim 19 , further comprising at least one of:
storing the exported word-list, in a form of binary serialization; and
compressing and storing the exported word-list, by using a compression algorithm.
21. The method of claim 1 , further comprising:
combining the text processing acceleration operator and a transformer encoder operator into one operator, to export a calculation diagram of a natural language understanding model.
22. The method of claim 1 , further comprising:
generating a framework calculation diagram of cyclic decoding, according to a decoding strategy;
embedding a transformer decoder operator into the framework calculation diagram of the cyclic decoding; and
exporting a calculation diagram of a natural language generating model.
23. The method of claim 1 , further comprising:
exporting a combined calculation diagram of natural language processing, the combined calculation diagram of the natural language processing comprising a calculation diagram combined by a calculation diagram of a natural language understanding model and a calculation diagram of a natural language generating model;
wherein the calculation diagram of the natural language understanding model comprises the text processing acceleration operator and a transformer encoder operator; and the calculation diagram of the natural language generating model comprises a transformer decoder operator and a decoding strategy operator.
24. The method of claim 1 , further comprising:
importing, at least one of a calculation diagram of a natural language understanding model, a calculation diagram of a natural language generating model, or a combined calculation diagram, into a deep learning framework of a plurality of devices.
25. An electronic device, comprising:
at least one processor; and
a memory connected in communication with the at least one processor;
wherein the memory stores an instruction executable by the at least one processor, and the instruction, when executed by the at least one processor, enables the at least one processor to execute the method of claim 1 .
26. A non-transitory computer-readable storage medium storing a computer instruction thereon, wherein the computer instruction is used to cause a computer to execute the method of claim 1 .