Neural text-to-speech synthesis with multi-level text information

1. A method for generating speech through neural text-to-speech (TTS) synthesis, comprising:

obtaining a text input having a sequence;

generating phoneme or character level text information based on the text input;

generating context-sensitive text information among words based on the text input, the context-sensitive text information having word level text information where generating the context-sensitive text information comprises:

identifying a word sequence from the text input;

up-sampling the word sequence to align with the text input;

generating a word embedding vector sequence of the word sequence and a phoneme vector sequence;

generating sentence level text information having a grammatical parsing information sequence, wherein generating the sentence level text information comprises:

performing grammatical parsing on the text input to obtain a grammatical structure of the text input; and

generating the grammatical parsing information sequence based on the grammatical structure of the text input by:

extracting grammatical parsing information of each word in the text input from the grammatical structure;

up-sampling the grammatical parsing information of each word to align with corresponding phonemes or characters in a phoneme or character sequence of the text input thereby generating a grammatical parsing information sequence; and

combining a phoneme vector sequence, the word embedding vector sequence and the grammatical parsing information sequence;

generating a text feature via a multi-input encoder coupled to receive the phoneme or character level text information and the word embedding vector sequence as inputs;

generating acoustic features from the text feature via a decoder; and

generating a speech waveform corresponding to the text input based at least on the text feature.

2. The method of claim 1 , wherein word embedding vector is generated with a word embedding model that is based on neural machine translation (NMT).

3. The method of claim 1 , wherein the generating the text feature comprises:

generating the text feature based on the phoneme or character level text information and the word level text information.

4. The method of claim 1 , wherein the grammatical parsing information of each word comprises at least one of:

an indication of phrase type of at least one phrase containing a word;

an indication of whether the word is a border of the at least one phrase; and

an indication of relative position of the word in the at least one phrase.

5. The method of claim 1 , wherein the generating the text feature comprises:

generating the text feature based on the phoneme or character level text information and the sentence level text information.

6. The method of claim 1 , wherein the generating the text feature comprises:

generating the text feature based on the phoneme or character level text information, the word level text information and the sentence level text information.

7. The method of claim 1 , wherein the generating the text feature comprises:

generating a first text feature based on the phoneme or character level text information through a first neural network;

generating at least one second text feature based on the word level text information and/or the sentence level text information comprised in the context-sensitive text information through at least one second neural network; and

generating the text feature through concatenating the first text feature with the at least one second text feature.

8. The method of claim 1 , wherein the generating the text feature comprises:

concatenating the phoneme or character level text information with the word level text information and/or the sentence level text information comprised in the context-sensitive text information, to form mixed text information; and

generating the text feature based on the mixed text information through a first neural network.

9. The method of claim 1 , wherein the generating the text feature comprises:

generating at least one compressed representation of the word level text information and/or the sentence level text information comprised in the context-sensitive text information through at least one first neural network;

concatenating the phoneme or character level text information with the at least one compressed representation to form mixed text information; and

generating the text feature based on the mixed text information through a second neural network.

10. An apparatus for generating speech through neural text-to-speech (TTS) synthesis, comprising:

a text input obtaining module, for obtaining a text input having a sequence;

a phoneme or character level text information generating module, for generating phoneme or character level text information based on the text input;

a context-sensitive text information generating module, for generating context-sensitive text information among words based on the text input, the context-sensitive text information having word level text information where generating the context-sensitive text information comprises:

identifying a word sequence from the text input; and

up-sampling the word sequence to align with the text input; generating a word embedding vector sequence of the word sequence and a phoneme vector sequence;

generating sentence level text information having a grammatical parsing information sequence, wherein generating the sentence level text information comprises:

performing grammatical parsing on the text input to obtain a grammatical structure of the text input; and

generating the grammatical parsing information sequence based on the grammatical structure of the text input by:

extracting grammatical parsing information of each word in the text input from the grammatical structure;

up-sampling the grammatical parsing information of each word to align with corresponding phonemes or characters in a phoneme or character sequence of the text input thereby generating a grammatical parsing information sequence; and

combining a phoneme vector sequence, the word embedding vector sequence and the grammatical parsing information sequence;

a text feature generating module comprising a multi-input encoder, for generating a text feature from inputs including the phoneme or character level text information and the word embedding vector sequence as inputs;

an acoustic features generating module comprising a decoder for generating acoustic features from the text feature; and

a speech waveform generating module, for generating a speech waveform corresponding to the text input based at least on the text feature.

11. An apparatus for generating speech through neural text-to-speech (TTS) synthesis, comprising:

at least one processor; and

a memory storing computer-executable instructions that, when executed, cause the at least one processor to:

obtain a text input having a sequence;

generate phoneme or character level text information and context-sensitive text information among words based on the text input, the context-sensitive text information having word level text information where generating the context-sensitive text information comprises:

identifying a word sequence from the text input;

up-sampling the word sequence to align with the text input;

generating a word embedding vector sequence of the word sequence and a phoneme vector sequence;

generating sentence level text information having a grammatical parsing information sequence, wherein generating the sentence level text information comprises:

performing grammatical parsing on the text input to obtain a grammatical structure of the text input; and

generating the grammatical parsing information sequence based on the grammatical structure of the text input by:

extracting grammatical parsing information of each word in the text input from the grammatical structure;

up-sampling the grammatical parsing information of each word to align with corresponding phonemes or characters in a phoneme or character sequence of the text input thereby generating a grammatical parsing information sequence; and

combining a phoneme vector sequence, the word embedding vector sequence and the grammatical parsing information sequence;

generate context-sensitive text information among words based on the text input;

generate a text feature via a multi-input encoder coupled to receive the phoneme or character level text information and the word embedding vector sequence as inputs;

generate acoustic features from the text feature via a decoder, and

generate a speech waveform corresponding to the text input based at least on the text feature.