Speech translation method, device, and storage medium
Provided are a speech translation method, a device, and a storage medium. The method includes: extracting, through an encoder of an end-to-end speech translation model, the semantic feature of a to-be-processed speech; decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature; decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature; and splitting the text sequence to obtain a target language text corresponding to the to-be-processed speech.
1 . A speech translation method, comprising:
extracting, through an encoder of an end-to-end speech translation model, a semantic feature of a to-be-processed speech;
decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature;
decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature, wherein the text sequence comprises the source language text and a target language text corresponding to the source language text; and
splitting the text sequence to obtain the target language text corresponding to the to-be-processed speech,
wherein a training process of the end-to-end speech translation model comprises:
pre-training, according to a text translation sample, the decoder of the end-to-end speech translation model to obtain an initial decoder; and
initializing the decoder of the end-to-end speech translation model based on the initial decoder; and training the initialized end-to-end speech translation model,
wherein the text translation sample comprises a source language sample text and a target language sample text corresponding to the source language sample text, and
wherein pre-training, according to the text translation sample, the decoder of the end-to-end speech translation model to obtain the initial decoder comprises:
splicing the source language sample text and the target language sample text to obtain a spliced sample sequence; and
pre-training, based on a masked cross-entropy loss function, the decoder by using the source language sample text and an all-zero vector as input of the decoder of the end-to-end speech translation model and the spliced sample sequence as desired output to obtain the initial decoder, wherein the masked cross-entropy loss function is configured to mask a prediction loss of a source language prediction text corresponding to the all-zero vector.
2 . The method according to claim 1 , wherein the encoder comprises a first encoder and a second encoder, and extracting, through the encoder of the end-to-end speech translation model, the semantic feature of the to-be-processed speech comprises:
extracting, through the first encoder, an acoustic feature of the to-be-processed speech;
detecting a blank frame and a duplicate frame in the acoustic feature, eliminating the blank frame and combining the duplicate frame to obtain a contracted acoustic feature; and
extracting, through the second encoder, the semantic feature in the contracted acoustic feature.
3 . The method according to claim 2 , wherein detecting the blank frame and the duplicate frame in the acoustic feature comprises:
detecting, based on a spike characteristic of a probability distribution of a connectionist temporal classification loss function, the blank frame and the duplicate frame in the acoustic feature.
4 . An electronic device, comprising a processor and a memory storing computer programs, wherein the computer programs, when executed by the processor, implement the following:
extracting, through an encoder of an end-to-end speech translation model, a semantic feature of a to-be-processed speech;
decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature;
decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature, wherein the text sequence comprises the source language text and a target language text corresponding to the source language text; and
splitting the text sequence to obtain the target language text corresponding to the to-be-processed speech,
wherein a training process of the end-to-end speech translation model comprises:
pre-training, according to a text translation sample, the decoder of the end-to-end speech translation model to obtain an initial decoder; and
initializing the decoder of the end-to-end speech translation model based on the initial decoder; and training the initialized end-to-end speech translation model,
wherein the text translation sample comprises a source language sample text and a target language sample text corresponding to the source language sample text, and
wherein pre-training, according to the text translation sample, the decoder of the end-to-end speech translation model to obtain the initial decoder comprises:
splicing the source language sample text and the target language sample text to obtain a spliced sample sequence; and
pre-training, based on a masked cross-entropy loss function, the decoder by using the source language sample text and an all-zero vector as input of the decoder of the end-to-end speech translation model and the spliced sample sequence as desired output to obtain the initial decoder, wherein the masked cross-entropy loss function is configured to mask a prediction loss of a source language prediction text corresponding to the all-zero vector.
5 . A non-transitory computer-readable storage medium storing computer programs, wherein the computer programs, when executed by a processor, implement the following:
extracting, through an encoder of an end-to-end speech translation model, a semantic feature of a to-be-processed speech;
decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature;
decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature, wherein the text sequence comprises the source language text and a target language text corresponding to the source language text; and
splitting the text sequence to obtain the target language text corresponding to the to-be-processed speech,
wherein a training process of the end-to-end speech translation model comprises:
pre-training, according to a text translation sample, the decoder of the end-to-end speech translation model to obtain an initial decoder; and
initializing the decoder of the end-to-end speech translation model based on the initial decoder; and training the initialized end-to-end speech translation model,
wherein the text translation sample comprises a source language sample text and a target language sample text corresponding to the source language sample text, and
wherein pre-training, according to the text translation sample, the decoder of the end-to-end speech translation model to obtain the initial decoder comprises:
splicing the source language sample text and the target language sample text to obtain a spliced sample sequence; and
pre-training, based on a masked cross-entropy loss function, the decoder by using the source language sample text and an all-zero vector as input of the decoder of the end-to-end speech translation model and the spliced sample sequence as desired output to obtain the initial decoder, wherein the masked cross-entropy loss function is configured to mask a prediction loss of a source language prediction text corresponding to the all-zero vector.
6 . The method according to claim 2 , wherein a training process of the first encoder comprises:
training, based on a connectionist temporal classification loss function, the first encoder by using a sample speech in a training sample set of the end-to-end speech translation model as input of the first encoder and a sample phoneme sequence corresponding to the sample speech as desired output.
7 . The electronic device according to claim 4 , wherein the encoder comprises a first encoder and a second encoder, and the computer programs implement extracting, through the encoder of the end-to-end speech translation model, the semantic feature of the to-be-processed speech by:
extracting, through the first encoder, an acoustic feature of the to-be-processed speech;
detecting a blank frame and a duplicate frame in the acoustic feature, eliminating the blank frame and combining the duplicate frame to obtain a contracted acoustic feature; and
extracting, through the second encoder, the semantic feature in the contracted acoustic feature.
8 . The electronic device according to claim 7 , wherein the computer programs implement detecting the blank frame and the duplicate frame in the acoustic feature by:
detecting, based on a spike characteristic of a probability distribution of a connectionist temporal classification loss function, the blank frame and the duplicate frame in the acoustic feature.
9 . The storage medium according to claim 5 , wherein the encoder comprises a first encoder and a second encoder, and the computer programs implement extracting, through the encoder of the end-to-end speech translation model, the semantic feature of the to-be-processed speech by:
extracting, through the first encoder, an acoustic feature of the to-be-processed speech;
detecting a blank frame and a duplicate frame in the acoustic feature, eliminating the blank frame and combining the duplicate frame to obtain a contracted acoustic feature; and
extracting, through the second encoder, the semantic feature in the contracted acoustic feature.
10 . The electronic device according to claim 4 , wherein a training process of the first encoder comprises:
training, based on a connectionist temporal classification loss function, the first encoder by using a sample speech in a training sample set of the end-to-end speech translation model as input of the first encoder and a sample phoneme sequence corresponding to the sample speech as desired output.
11 . The storage medium according to claim 9 , wherein detecting the blank frame and the duplicate frame in the acoustic feature comprises:
detecting, based on a spike characteristic of a probability distribution of a connectionist temporal classification loss function, the blank frame and the duplicate frame in the acoustic feature.
12 . The storage medium according to claim 5 , wherein a training process of the first encoder comprises:
training, based on a connectionist temporal classification loss function, the first encoder by using a sample speech in a training sample set of the end-to-end speech translation model as input of the first encoder and a sample phoneme sequence corresponding to the sample speech as desired output.