Method and system for diacritizing Arabic text
The presently disclosed method and system automatically diacritize written Arabic text for use with applications that require verbalizing Arabic text. A method may comprise converting a written sentence into a word sequence and identifying a target source word. The method then may comprise repeatedly overlaying and translating a context window at a plurality of positions in the word sequence to select a plurality of subsets of the word sequence contained within the context window. A diacritized form of the target source word may be generated in each of the word sequence subsets. A final diacritized form of the target source word may be selected from the plurality of diacritized forms based on a voting scheme. The voting scheme may include selecting the diacritized form that is generated the most or may be based on a system of weighting factors.
1 . A method comprising:
(A) receiving a source sentence;
(B) converting the source sentence into a word sequence, wherein the word sequence comprises a plurality of words including a target source word;
(C) overlaying a context window at a first position within the word sequence to select a first subset of the word sequence contained within the context window at the first position, wherein the first subset of the word sequence contains the target source word;
(D) generating a first diacritized form of the target source word based on the first subset of the word sequence;
(E) translating the context window to a second position within the word sequence to select a second subset of the word sequence contained within the context window at the second position, wherein the second subset of the word sequence contains the target source word;
(F) generating a second diacritized form of the target source word in the second subset of the word sequence;
(G) repeating steps (E) through (F) to generate a plurality of diacritized forms of the target source word; and
(H) selecting a final diacritized form of the target source word from the plurality of diacritized forms of the target source word according to a voting scheme, wherein the context window is a varying number of words in length and the voting scheme weights each diacritized form of the plurality of diacritized forms based on a given length of the context window used to generate a corresponding diacritized form.
2 . The method of claim 1 , wherein each of the plurality of words of the word sequence comprises a sequence of characters.
3 . The method of claim 2 , wherein the sequence of characters comprises a character boundary in between each character within each of the plurality of words and a word boundary between each of the plurality of words in the word sequence.
4 . The method of claim 1 , wherein the voting scheme further comprises, responsive to identifying more than one diacritized form of the target source word generated the most number of times, selecting the diacritized form of the target source word from the segment of words in which the target source word is the center word of the segment of words.
5 . The method of claim 1 , wherein the context window is a fixed number of words in length.
6 . The method of claim 1 , wherein the context window is a fixed number of characters in length.
7 . The method of claim 1 , wherein generating a diacritized form of a target source word comprises inputting the word sequence into at least one machine learning model.
8 . The method of claim 7 , wherein each of the plurality of diacritized forms is generated by the same machine learning model.
9 . The method of claim 7 , wherein the plurality of diacritized forms are generated by more than one machine learning model.
10 . The method of claim 7 , wherein the at least one machine learning model is a sequence to sequence model.
11 . The method of claim 1 , wherein translating the context window further comprises incrementing the position of the context window by a set number of words.
12 . The method of claim 11 , wherein the set number of words is one word.
13 . A system comprising:
a processor; and
a memory storing instructions which, when executed by the processor, cause the processor to:
(A) receive a source sentence;
(B) convert the source sentence into a word sequence, wherein the word sequence comprises a plurality of words including a target source word;
(C) overlay a context window at a first position within the word sequence to select a first subset of the word sequence contained within the context window at the first position, wherein the first subset of the word sequence contains the target source word;
(D) generate a first diacritized form of the target source word based on the first subset of the word sequence;
(E) translate the context window to a second position within the word sequence to select a second subset of the word sequence contained within the context window at the second position, wherein the second subset of the word sequence contains the target source word;
(F) generate a second diacritized form of the target source word in the second subset of the word sequence;
(G) repeat steps (E) through (F) to generate a plurality of diacritized forms of the target source word; and
(H) select a final diacritized form of the target source word from the plurality of diacritized forms of the target source word according to a voting scheme, wherein the context window is a varying number of words in length and the voting scheme weights each diacritized form of the plurality of diacritized forms based on a given length of the context window used to generate a corresponding diacritized form.
14 . The system of claim 13 , wherein each of the plurality of words of the word sequence comprises a sequence of characters.
15 . The system of claim 13 , wherein the context window is a varying number of words in length and the voting scheme comprises selecting the diacritized form of the target source word based on weighting factors assigned to each respective context window of a given length.
16 . A non-transitory, computer-readable medium storing instructions which, when performed by a processor, cause the processor to:
(A) receive a source sentence;
(B) convert the source sentence into a word sequence, wherein the word sequence comprises a plurality of words including a target source word;
(C) overlay a context window at a first position within the word sequence to select a first subset of the word sequence contained within the context window at the first position, wherein the first subset of the word sequence contains the target source word;
(D) generate a first diacritized form of the target source word based on the first subset of the word sequence;
(E) translate the context window to a second position within the word sequence to select a second subset of the word sequence contained within the context window at the second position, wherein the second subset of the word sequence contains the target source word;
(F) generate a second diacritized form of the target source word in the second subset of the word sequence;
(G) repeat steps (E) through (F) to generate a plurality of diacritized forms of the target source word; and
(H) select a final diacritized form of the target source word from the plurality of diacritized forms of the target source word according to a voting scheme, wherein the context window is a varying number of words in length and the voting scheme weights each diacritized form of the plurality of diacritized forms based on a given length of the context window used to generate a corresponding diacritized form.