Granted Patent B2

US 12,387,710 · App. 17/965,130 · Granted Aug 12, 2025

Inventors: Shilun Lin (Shenzhen, CN); Xinhui Li (Shenzhen, CN); Li Lu (Shenzhen, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G10L13/02G10L13/047G10L13/08

Abstract

Claims (95)

1. An audio processing method, executed by an electronic device, comprising:

performing speech feature conversion on a text to obtain at least one acoustic feature frame;

extracting a conditional feature corresponding to each acoustic feature frame from each acoustic feature frame of the at least one acoustic feature frame by a frame rate network;

performing frequency division and time-domain down-sampling on the current frame of each acoustic feature frame to obtain n subframes corresponding to the current frame, n being a positive integer greater than 1, and each subframe of the n subframes comprising a preset number of sampling points;

synchronously predicting, by a sampling prediction network, in an ith prediction process, sample values corresponding to current m adjacent sampling points on the n subframes to obtain m×n sub-prediction values, and obtain n sub-prediction values corresponding to each sampling point of the preset number of sampling points, i being a positive integer greater than or equal to 1, and m being a positive integer greater than or equal to 2 and less than or equal to the preset number;

obtaining an audio prediction signal corresponding to the current frame according to the n sub-prediction values corresponding to each sampling point; and

performing audio synthesis on the audio prediction signal corresponding to each acoustic feature frame of the at least one acoustic feature frame to obtain a target audio corresponding to the text.

2. The method according to claim 1 , wherein when m equals to 2, the sampling prediction network comprises 2n independent fully connected layers, and the two adjacent sampling points comprise: in the ith prediction process, sampling point t corresponding to the current time t and sampling point t+1 corresponding to the next time t+1, t being a positive integer greater than or equal to 1;

the synchronously predicting sample values corresponding to the current m adjacent sampling points on the n subframes to obtain m×n sub-prediction values, comprises:

in the ith prediction process, based on at least one historical sampling point at time t corresponding to the sampling point t, performing linear coding prediction, by the sampling prediction network, on linear sample values of the sampling point t on the n subframes, to obtain n sub-rough prediction values at time t;

when i is greater than 1, based on a historical prediction result corresponding to the (i−1)th prediction process, and combined with the conditional features, by 2n fully connected layers, performing forward residual prediction synchronously on residuals of the sampling point t and residuals of the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1, the historical prediction result comprising n residuals and n sub-prediction values corresponding to each of two adjacent sampling points in the (i−1)th prediction process;

performing linear coding prediction on linear sampling values of the sampling point t+1 on the n subframes to obtain n sub-rough prediction values at time t+1 based on at least one historical sampling point at time t+1 corresponding to the sampling point t+1;

obtaining n sub-prediction values at time t corresponding to the sampling point t according to the n residuals at time t and the n sub-rough prediction values at time t, and obtaining n sub-prediction values at time t+1 according to the n residuals at time t+1 and the n sub-rough prediction values at time t+1; and using the n sub-prediction values at time t and the n sub-prediction values at time t+1 as 2n sub-prediction values.

3. The method according to claim 2 , wherein the based on a historical prediction result corresponding to the (i−1)th prediction process, and combined with the conditional features, by 2n fully connected layers, performing forward residual prediction synchronously on residuals of the sampling point t and residuals of the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1, comprises:

obtaining n sub-rough prediction values at time t−1 corresponding to the sampling point t−1, as well as n residuals at time t−1, n residuals at time t−2, n sub-prediction values at time t−1 and n prediction values at time t−2 in the (i−1)th prediction process;

performing feature dimension filtering on the n sub-rough prediction values at time t, the n sub-rough prediction values at time t−1, the n residuals at time t−1, the n residuals at time t−2, the n sub-prediction values at time t−1 and the n prediction values at time t−2, to obtain a dimension reduced feature set; and

synchronously performing forward residual prediction on residuals of the sampling point t and the sampling point t+1 on each subframe of the n subframes respectively, by each fully connected layer of the 2n fully connected layers, combined with the conditional features, and based on the dimension reduced feature set, to obtain n residuals at time t and n residuals at time t+1 respectively.

4. The method according to claim 3 , wherein the by each fully connected layer of the 2n fully connected layers, combined with the conditional features, and based on the dimension reduced feature set, synchronously performing forward residual prediction on residuals of the sampling point t and the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t and n residuals at time t+1 respectively, comprises:

determining n dimension reduction residuals at time t−2 and n dimension reduced prediction values at time t−2 in the dimension reduced feature set as excitation values at time t, the n dimension reduction residuals at time t−2 being obtained by performing feature dimension filtering on n residuals at time t−2, and the n dimension reduced prediction values at time t−2 being obtained by performing feature dimension filtering on n prediction values at time t−2;

determining the n dimension reduction residuals at time t−1 and the n dimension reduced prediction values at time t−1 in the dimension reduced feature set as excitation values at time t+1, the n dimension reduction residuals at time t−1 being obtained by performing feature dimension filtering on n residuals at time t−1, and the n dimension reduced prediction values at time t−1 being obtained by performing feature dimension filtering on n prediction values at time t−1;

performing forward residual prediction on the sampling point t according to the n dimension reduced sub-rough prediction values at time t−1 to obtain the n residuals at time t in n fully connected layers of the 2n fully connected layers, based on the conditional features and the excitation values at time t, by each fully connected layer in the n fully connected layers; and

performing forward residual prediction on the sampling point t+1 according to the n dimension reduced sub-rough prediction values at time t, to obtain the n residuals at time t+1 in the other n fully connected layers of the 2n fully connected layers, based on the conditional features and the excitation values at time t+1, by each fully connected layer in the other n fully connected layers.

5. The method according to claim 3 , wherein the sampling prediction network comprises a first gated recurrent network and a second gated recurrent network; and the performing feature dimension filtering on the n sub-rough prediction values at time t, the n sub-rough prediction values at time t−1, the n residuals at time t−1, the n residuals at time t−2, the n sub-prediction values at time t−1 and the n prediction values at time t−2, to obtain a dimension reduced feature set, comprises:

performing feature dimension merge on the n sub-rough prediction values at time t, the n sub-rough prediction values at time t−1, the n residuals at time t−1, the n residuals at time t−2, the n sub-prediction values at time t−1, and the n prediction values at time t−2 to obtain an initial feature vector set;

performing feature dimension reduction on the initial feature vector set by the first gated recurrent network to obtain an intermediate feature vector set based on the conditional features; and

performing feature dimension reduction on the intermediate feature vector set by the second gated recurrent network to obtain the dimension reduced feature set based on the conditional features.

6. The method according to claim 2 , further comprising:

when t is less than or equal to a preset window threshold, using all sampling points before the sampling point t as the at least one historical sampling point at time t, the preset window threshold representing the maximum quantity of sampling points processible by linear coding prediction; or

when t is greater than the preset window threshold, using sampling points in a range of the sampling point t−1 to sampling point t−k, as the at least one historical sampling point at time t, k being the preset window threshold.

7. The method according to claim 2 , further comprising:

when i is equal to 1, by 2n fully connected layers, combined with the conditional features and preset excitation parameters, performing forward residual prediction on residuals of the sampling point t and the sampling point t+1 on the n subframes synchronously, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1;

based on at least one historical sampling point at time t+1 corresponding to the sampling point t+1, performing linear coding prediction on linear sampling values of the sampling point t+1 on the n subframes to obtain n sub-rough prediction values at time t+1; and

obtaining n sub-prediction values at time t corresponding to the sampling point t according to the n residuals at time t and the n sub-rough prediction values at time t, and obtaining n sub-prediction values at time t+1 according to the n residuals at time t+1 and the n sub-rough prediction values at time t+1; and using the n sub-prediction values at time t and the n sub-prediction values at time t+1 as the 2n sub-prediction values.

8. The method according to claim 1 , wherein the performing frequency division and time-domain down-sampling on the current frame of each acoustic feature frame to obtain n subframes corresponding to the current frame, comprises:

performing frequency-domain division on the current frame to obtain n initial subframes; and

down-sampling time-domain sampling points corresponding to the n initial subframes to obtain the n subframes.

9. The method according to claim 1 , wherein the obtaining an audio prediction signal corresponding to the current frame according to the n sub-prediction values corresponding to each sampling point, and performing audio synthesis on the audio prediction signal corresponding to each acoustic feature frame of the at least one acoustic feature frame to obtain a target audio corresponding to the text, comprises:

superposing the n sub-prediction values corresponding to each sampling point in the frequency domain to obtain the signal prediction value corresponding to each sampling point;

performing time-domain signal synthesis on the signal prediction values corresponding to each sampling point to obtain an audio prediction signal corresponding to the current frame, and obtain an audio signal corresponding to each frame of acoustic feature; and

performing signal synthesis on the audio signal corresponding to each frame of acoustic feature to obtain the target audio.

10. The method according to claim 1 , wherein the performing speech feature conversion on a text to obtain at least one acoustic feature frame, comprises:

acquiring a text;

preprocessing the text to obtain text information; and

performing acoustic feature prediction on the text information by a text-to-speech conversion model to obtain the at least one acoustic feature frame, sub-predictionsub-predictionsub-predictionsub-predictionsub-predictionsub-prediction.

11. An electronic device, comprising:

a memory, configured to store executable instructions; and

a processor, when executing the executable instructions stored in the memory, configured to implement:

performing speech feature conversion on a text to obtain at least one acoustic feature frame;

extracting a conditional feature corresponding to each acoustic feature frame from each acoustic feature frame of the at least one acoustic feature frame by a frame rate network;

performing frequency division and time-domain down-sampling on the current frame of each acoustic feature frame to obtain n subframes corresponding to the current frame, n being a positive integer greater than 1, and each subframe of the n subframes comprising a preset number of sampling points;

synchronously predicting, by a sampling prediction network, in an ith prediction process, sample values corresponding to current m adjacent sampling points on the n subframes to obtain m×n sub-prediction values, and obtain n sub-prediction values corresponding to each sampling point of the preset number of sampling points, i being a positive integer greater than or equal to 1, and m being a positive integer greater than or equal to 2 and less than or equal to the preset number;

obtaining an audio prediction signal corresponding to the current frame according to the n sub-prediction values corresponding to each sampling point; and

performing audio synthesis on the audio prediction signal corresponding to each acoustic feature frame of the at least one acoustic feature frame to obtain a target audio corresponding to the text.

12. The electronic device according to claim 11 , wherein when m equals to 2, the sampling prediction network comprises 2n independent fully connected layers, and the two adjacent sampling points comprise: in the ith prediction process, sampling point t corresponding to the current time t and sampling point t+1 corresponding to the next time t+1, t being a positive integer greater than or equal to 1;

the synchronously predicting sample values corresponding to the current m adjacent sampling points on the n subframes to obtain m×n sub-prediction values, comprises:

in the ith prediction process, based on at least one historical sampling point at time t corresponding to the sampling point t, performing linear coding prediction, by the sampling prediction network, on linear sample values of the sampling point t on the n subframes, to obtain n sub-rough prediction values at time t;

when i is greater than 1, based on a historical prediction result corresponding to the (i−1)th prediction process, and combined with the conditional features, by 2n fully connected layers, performing forward residual prediction synchronously on residuals of the sampling point t and residuals of the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1, the historical prediction result comprising n residuals and n sub-prediction values corresponding to each of two adjacent sampling points in the (i−1)th prediction process;

performing linear coding prediction on linear sampling values of the sampling point t+1 on the n subframes to obtain n sub-rough prediction values at time t+1 based on at least one historical sampling point at time t+1 corresponding to the sampling point t+1;

obtaining n sub-prediction values at time t corresponding to the sampling point t according to the n residuals at time t and the n sub-rough prediction values at time t, and obtaining n sub-prediction values at time t+1 according to the n residuals at time t+1 and the n sub-rough prediction values at time t+1; and using the n sub-prediction values at time t and the n sub-prediction values at time t+1 as 2n sub-prediction values.

13. The electronic device according to claim 12 , wherein the based on a historical prediction result corresponding to the (i−1)th prediction process, and combined with the conditional features, by 2n fully connected layers, performing forward residual prediction synchronously on residuals of the sampling point t and residuals of the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1, comprises:

obtaining n sub-rough prediction values at time t−1 corresponding to the sampling point t−1, as well as n residuals at time t−1, n residuals at time t−2, n sub-prediction values at time t−1 and n prediction values at time t−2 in the (i−1)th prediction process;

performing feature dimension filtering on the n sub-rough prediction values at time t, the n sub-rough prediction values at time t−1, the n residuals at time t−1, the n residuals at time t−2, the n sub-prediction values at time t−1 and the n prediction values at time t−2, to obtain a dimension reduced feature set; and

synchronously performing forward residual prediction on residuals of the sampling point t and the sampling point t+1 on each subframe of the n subframes respectively, by each fully connected layer of the 2n fully connected layers, combined with the conditional features, and based on the dimension reduced feature set, to obtain n residuals at time t and n residuals at time t+1 respectively.

14. The electronic device according to claim 13 , wherein the by each fully connected layer of the 2n fully connected layers, combined with the conditional features, and based on the dimension reduced feature set, synchronously performing forward residual prediction on residuals of the sampling point t and the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t and n residuals at time t+1 respectively, comprises:

determining n dimension reduction residuals at time t−2 and n dimension reduced prediction values at time t−2 in the dimension reduced feature set as excitation values at time t, the n dimension reduction residuals at time t−2 being obtained by performing feature dimension filtering on n residuals at time t−2, and the n dimension reduced prediction values at time t−2 being obtained by performing feature dimension filtering on n prediction values at time t−2;

determining the n dimension reduction residuals at time t−1 and the n dimension reduced prediction values at time t−1 in the dimension reduced feature set as excitation values at time t+1, the n dimension reduction residuals at time t−1 being obtained by performing feature dimension filtering on n residuals at time t−1, and the n dimension reduced prediction values at time t−1 being obtained by performing feature dimension filtering on n prediction values at time t−1;

performing forward residual prediction on the sampling point t according to the n dimension reduced sub-rough prediction values at time t−1 to obtain the n residuals at time t in n fully connected layers of the 2n fully connected layers, based on the conditional features and the excitation values at time t, by each fully connected layer in the n fully connected layers; and

performing forward residual prediction on the sampling point t+1 according to the n dimension reduced sub-rough prediction values at time t, to obtain the n residuals at time t+1 in the other n fully connected layers of the 2n fully connected layers, based on the conditional features and the excitation values at time t+1, by each fully connected layer in the other n fully connected layers.

15. The electronic device according to claim 13 , wherein the sampling prediction network comprises a first gated recurrent network and a second gated recurrent network; and the performing feature dimension filtering on the n sub-rough prediction values at time t, the n sub-rough prediction values at time t−1, the n residuals at time t−1, the n residuals at time t−2, the n sub-prediction values at time t−1 and the n prediction values at time t−2, to obtain a dimension reduced feature set, comprises:

performing feature dimension merge on the n sub-rough prediction values at time t, the n sub-rough prediction values at time t−1, the n residuals at time t−1, the n residuals at time t−2, the n sub-prediction values at time t−1, and the n prediction values at time t−2 to obtain an initial feature vector set;

performing feature dimension reduction on the initial feature vector set by the first gated recurrent network to obtain an intermediate feature vector set based on the conditional features; and

performing feature dimension reduction on the intermediate feature vector set by the second gated recurrent network to obtain the dimension reduced feature set based on the conditional features.

16. The electronic device according to claim 11 , wherein the performing frequency division and time-domain down-sampling on the current frame of each acoustic feature frame to obtain n subframes corresponding to the current frame, comprises:

performing frequency-domain division on the current frame to obtain n initial subframes; and

down-sampling time-domain sampling points corresponding to the n initial subframes to obtain the n subframes.

17. A non-transitory computer-readable storage medium, storing executable instructions, and when executed by a processor, causing the processor to implement:

performing speech feature conversion on a text to obtain at least one acoustic feature frame;

extracting a conditional feature corresponding to each acoustic feature frame from each acoustic feature frame of the at least one acoustic feature frame by a frame rate network;

performing frequency division and time-domain down-sampling on the current frame of each acoustic feature frame to obtain n subframes corresponding to the current frame, n being a positive integer greater than 1, and each subframe of the n subframes comprising a preset number of sampling points;

synchronously predicting, by a sampling prediction network, in an ith prediction process, sample values corresponding to current m adjacent sampling points on the n subframes to obtain m×n sub-prediction values, and obtain n sub-prediction values corresponding to each sampling point of the preset number of sampling points, i being a positive integer greater than or equal to 1, and m being a positive integer greater than or equal to 2 and less than or equal to the preset number;

obtaining an audio prediction signal corresponding to the current frame according to the n sub-prediction values corresponding to each sampling point; and

performing audio synthesis on the audio prediction signal corresponding to each acoustic feature frame of the at least one acoustic feature frame to obtain a target audio corresponding to the text.

18. The computer-readable storage medium according to claim 17 , wherein when m equals to 2, the sampling prediction network comprises 2n independent fully connected layers, and the two adjacent sampling points comprise: in the ith prediction process, sampling point t corresponding to the current time t and sampling point t+1 corresponding to the next time t+1, t being a positive integer greater than or equal to 1;

the synchronously predicting sample values corresponding to the current m adjacent sampling points on the n subframes to obtain m×n sub-prediction values, comprises:

in the ith prediction process, based on at least one historical sampling point at time t corresponding to the sampling point t, performing linear coding prediction, by the sampling prediction network, on linear sample values of the sampling point t on the n subframes, to obtain n sub-rough prediction values at time t;

when i is greater than 1, based on a historical prediction result corresponding to the (i−1)th prediction process, and combined with the conditional features, by 2n fully connected layers, performing forward residual prediction synchronously on residuals of the sampling point t and residuals of the sampling point t+1 on each subframe of the n subframes respectively, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1, the historical prediction result comprising n residuals and n sub-prediction values corresponding to each of two adjacent sampling points in the (i−1)th prediction process;

performing linear coding prediction on linear sampling values of the sampling point t+1 on the n subframes to obtain n sub-rough prediction values at time t+1 based on at least one historical sampling point at time t+1 corresponding to the sampling point t+1;

obtaining n sub-prediction values at time t corresponding to the sampling point t according to the n residuals at time t and the n sub-rough prediction values at time t, and obtaining n sub-prediction values at time t+1 according to the n residuals at time t+1 and the n sub-rough prediction values at time t+1; and using the n sub-prediction values at time t and the n sub-prediction values at time t+1 as 2n sub-prediction values.

19. The computer-readable storage medium according to claim 18 , wherein the executable instructions further cause the processor to implement:

when t is less than or equal to a preset window threshold, using all sampling points before the sampling point t as the at least one historical sampling point at time t, the preset window threshold representing the maximum quantity of sampling points processible by linear coding prediction; or

when t is greater than the preset window threshold, using sampling points in a range of the sampling point t−1 to sampling point t−k, as the at least one historical sampling point at time t, k being the preset window threshold.

20. The computer-readable storage medium according to claim 18 , wherein the executable instructions further cause the processor to implement:

when i is equal to 1, by 2n fully connected layers, combined with the conditional features and preset excitation parameters, performing forward residual prediction on residuals of the sampling point t and the sampling point t+1 on the n subframes synchronously, to obtain n residuals at time t corresponding to the sampling point t and n residuals at time t+1 corresponding to the sampling point t+1;

based on at least one historical sampling point at time t+1 corresponding to the sampling point t+1, performing linear coding prediction on linear sampling values of the sampling point t+1 on the n subframes to obtain n sub-rough prediction values at time t+1; and

obtaining n sub-prediction values at time t corresponding to the sampling point t according to the n residuals at time t and the n sub-rough prediction values at time t, and obtaining n sub-prediction values at time t+1 according to the n residuals at time t+1 and the n sub-rough prediction values at time t+1; and using the n sub-prediction values at time t and the n sub-prediction values at time t+1 as the 2n sub-prediction values.

Assignments (2)

Priority Claims (1)

Continuity (2)

References Cited (34)