Audio processing method and apparatus based on artificial intelligence, device, storage medium, and computer program product
Disclosed are an audio processing method performed by an electronic device, a non-transitory computer-readable storage medium, and a computer program product. The method includes: sampling multiple fragments of audio data of a target object to obtain reference audio data of the target object; performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data; performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object, wherein the tone embedding vector is independent from content of the audio data; and generating audio data of a target text that conforms to a tone of the target object according to the tone embedding vector of the target object.
1. An audio processing method performed by an electronic device, the method comprising:
sampling multiple fragments of audio data of a target object to obtain reference audio data of the target object;
performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data;
performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object, wherein the tone embedding vector is independent from content of the audio data;
encoding a target text to obtain a content embedding vector of the target text;
aligning the tone embedding vector of the target object and the content embedding vector of the target text to obtain an aligned vector of the target text;
decoding the aligned vector of the target text to obtain an acoustic feature sequence of the target text; and
performing acoustic feature conversion on the acoustic feature sequence, to obtain audio data of the target text.
2. The method according to claim 1 , wherein the performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object comprises:
mapping, based on an embedding space of multiple object tones, the reference embedding vector of the reference audio data to obtain weights of the multiple object tones; and
performing, based on the weights of the multiple object tones, weighted summation on embedding vectors of the multiple object tones to obtain the tone embedding vector of the target object.
3. The method according to claim 1 , wherein the performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data comprises:
performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data; and
encoding the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.
4. The method according to claim 3 , wherein
the audio encoding is implemented by an encoder, the encoder comprising multiple cascaded convolutional layers; and
the performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data comprises:
performing, by a first convolutional layer among the multiple cascaded convolutional layers, convolution processing on the reference audio data of the target object; and
outputting a convolution result of the first convolutional layer to a subsequent cascaded convolutional layer, continuing, by the subsequent cascaded convolutional layer, to perform the convolution processing and output a convolution result until a last convolutional layer outputs a convolution result, and
using the convolution result output by the last convolutional layer as the convolution embedding vector of the reference audio data.
5. The method according to claim 3 , wherein
the audio encoding is implemented by an encoder, the encoder comprising a recurrent neural network; and
the encoding the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data comprises:
updating a hidden vector of the recurrent neural network and the convolution embedding vector of the reference audio data to obtain update information about the reference audio data;
resetting the hidden vector of the recurrent neural network and the convolution embedding vector of the reference audio data to obtain reset information about the reference audio data; and
performing, based on the update information about the reference audio data and the reset information about the reference audio data, context encoding on the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.
6. The method according to claim 1 , wherein the performing acoustic feature conversion on the acoustic feature sequence, to obtain the audio data of the target text comprises:
smoothing the acoustic feature sequence to obtain frequency spectrum data corresponding to the acoustic feature sequence; and
performing Fourier transform on the frequency spectrum data corresponding to the acoustic feature sequence, to obtain the audio data which conform to the tone of the target object and correspond to the target text.
7. An electronic device, comprising:
a memory, configured to store executable instructions; and
a processor, configured to, when executing the executable instructions stored in the memory, cause the electronic device to perform an audio processing method including:
sampling multiple fragments of audio data of a target object to obtain reference audio data of the target object;
performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data;
performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object, wherein the tone embedding vector is independent from content of the audio data;
encoding a target text to obtain a content embedding vector of the target text;
aligning the tone embedding vector of the target object and the content embedding vector of the target text to obtain an aligned vector of the target text;
decoding the aligned vector of the target text to obtain an acoustic feature sequence of the target text; and
performing acoustic feature conversion on the acoustic feature sequence, to obtain audio data of the target text.
8. The electronic device according to claim 7 , wherein the performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object comprises:
mapping, based on an embedding space of multiple object tones, the reference embedding vector of the reference audio data to obtain weights of the multiple object tones; and
performing, based on the weights of the multiple object tones, weighted summation on embedding vectors of the multiple object tones to obtain the tone embedding vector of the target object.
9. The electronic device according to claim 7 , wherein the performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data comprises:
performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data; and
encoding the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.
10. The electronic device according to claim 7 , wherein the performing acoustic feature conversion on the acoustic feature sequence, to obtain the audio data of the target text comprises:
smoothing the acoustic feature sequence to obtain frequency spectrum data corresponding to the acoustic feature sequence; and
performing Fourier transform on the frequency spectrum data corresponding to the acoustic feature sequence, to obtain the audio data which conform to the tone of the target object and correspond to the target text.
11. A non-transitory computer-readable storage medium storing executable instructions for, when executed by a processor of an electronic device, causing the electronic device to perform an audio processing method including:
sampling multiple fragments of audio data of a target object to obtain reference audio data of the target object;
performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data;
performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object, wherein the tone embedding vector is independent from content of the audio data;
encoding a target text to obtain a content embedding vector of the target text;
aligning the tone embedding vector of the target object and the content embedding vector of the target text to obtain an aligned vector of the target text;
decoding the aligned vector of the target text to obtain an acoustic feature sequence of the target text; and
performing acoustic feature conversion on the acoustic feature sequence, to obtain audio data of the target text.
12. The non-transitory computer-readable storage medium according to claim 11 , wherein the performing tone-based attention processing on the reference embedding vector of the reference audio data to obtain a tone embedding vector of the target object comprises:
mapping, based on an embedding space of multiple object tones, the reference embedding vector of the reference audio data to obtain weights of the multiple object tones; and
performing, based on the weights of the multiple object tones, weighted summation on embedding vectors of the multiple object tones to obtain the tone embedding vector of the target object.
13. The non-transitory computer-readable storage medium according to claim 11 , wherein the performing audio encoding on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data comprises:
performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data; and
encoding the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.
14. The non-transitory computer-readable storage medium according to claim 11 , wherein the performing acoustic feature conversion on the acoustic feature sequence, to obtain the audio data of the target text comprises:
smoothing the acoustic feature sequence to obtain frequency spectrum data corresponding to the acoustic feature sequence; and
performing Fourier transform on the frequency spectrum data corresponding to the acoustic feature sequence, to obtain the audio data which conform to the tone of the target object and correspond to the target text.
15. The electronic device according to claim 9 , wherein the audio encoding is implemented by an encoder, the encoder comprising multiple cascaded convolutional layers; and
the performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data comprises:
performing, by a first convolutional layer among the multiple cascaded convolutional layers, convolution processing on the reference audio data of the target object; and
outputting a convolution result of the first convolutional layer to a subsequent cascaded convolutional layer, continuing, by the subsequent cascaded convolutional layer, to perform the convolution processing and output a convolution result until a last convolutional layer outputs a convolution result, and
using the convolution result output by the last convolutional layer as the convolution embedding vector of the reference audio data.
16. The electronic device according to claim 9 , wherein the audio encoding is implemented by an encoder, the encoder comprising a recurrent neural network; and
the encoding the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data comprises:
updating a hidden vector of the recurrent neural network and the convolution embedding vector of the reference audio data to obtain update information about the reference audio data;
resetting the hidden vector of the recurrent neural network and the convolution embedding vector of the reference audio data to obtain reset information about the reference audio data; and
performing, based on the update information about the reference audio data and the reset information about the reference audio data, context encoding on the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.
17. The non-transitory computer-readable storage medium according to claim 13 , wherein the audio encoding is implemented by an encoder, the encoder comprising multiple cascaded convolutional layers; and
the performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data comprises:
performing, by a first convolutional layer among the multiple cascaded convolutional layers, convolution processing on the reference audio data of the target object; and
outputting a convolution result of the first convolutional layer to a subsequent cascaded convolutional layer, continuing, by the subsequent cascaded convolutional layer, to perform the convolution processing and output a convolution result until a last convolutional layer outputs a convolution result, and
using the convolution result output by the last convolutional layer as the convolution embedding vector of the reference audio data.
18. The non-transitory computer-readable storage medium according to claim 13 , wherein the audio encoding is implemented by an encoder, the encoder comprising a recurrent neural network; and
the encoding the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data comprises:
updating a hidden vector of the recurrent neural network and the convolution embedding vector of the reference audio data to obtain update information about the reference audio data;
resetting the hidden vector of the recurrent neural network and the convolution embedding vector of the reference audio data to obtain reset information about the reference audio data; and
performing, based on the update information about the reference audio data and the reset information about the reference audio data, context encoding on the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.