Granted Patent B2

US 12,277,953 · App. 17/581,537 · Granted Apr 15, 2025

Inventors: Wei Xiao (Guangdong, CN); Meng Wang (Guangdong, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G10L21/034G10L15/063G10L25/18G10L25/30

Abstract

Claims (95)

1. A speech signal processing method, comprising:

obtaining speech feature parameters of a to-be-processed speech signal, the speech feature parameters including an initial spectrum of the to-be-processed speech signal;

predicting, by processing circuitry of an electronic device, a first quantity of subband spectrum envelopes based on the speech feature parameters, the first quantity being less than a quantity of initial spectrum coefficients included in the initial spectrum, and each of the subband spectrum envelopes corresponding to a second quantity of initial spectrum coefficients;

dividing the initial spectrum into a first quantity of spectrum subbands, each of the spectrum subbands including the second quantity of initial spectrum coefficients in the initial spectrum;

obtaining a respective to-be-processed subband spectrum envelope for each spectrum subband based on the initial spectrum coefficients included in the respective spectrum subband;

determining a respective difference value for each of the predicted subband spectrum envelopes between the respective predicted subband spectrum envelope and a corresponding to-be-processed subband spectrum envelope;

adjusting the initial spectrum coefficients for each to-be-processed subband spectrum envelope, based on the difference value corresponding to the respective to-be-processed subband spectrum envelope, to obtain an adjusted spectrum; and

obtaining an enhanced speech signal corresponding to the to-be-processed speech signal based on the adjusted spectrum.

2. The method according to claim 1 , wherein the speech feature parameters further include time domain feature parameters of the to-be-processed speech signal.

3. The method according to claim 2 , wherein the time domain feature parameters include a line spectral frequency (LSF) parameter.

4. The method according to claim 3 , wherein the to-be-processed speech signal is of a current speech frame of multiple speech frames of a to-be-enhanced speech signal, and the LSF parameter of the current speech frame includes at least one of (i) an angular frequency shift of the current speech frame relative to an adjacent speech frame of the current speech frame, (ii) an average angular frequency of the current speech frame, or (iii) a deviation of an angular frequency distribution of the current speech frame relative to a reference angular frequency distribution.

5. The method according to claim 4 , wherein each of the multiple speech frames includes at least two subframes, and the LSF parameter of each speech frame includes the LSF parameter of at least one subframe included in the respective speech frame.

6. The method according to claim 1 , wherein both the predicted subband spectrum envelopes and the to-be-processed subband spectrum envelopes are in a logarithm domain, and for one to-be-processed subband spectrum envelope, the adjusting the initial spectrum coefficients comprises:

respectively performing a logarithmic transformation on each of the initial spectrum coefficients corresponding to the to-be-processed subband spectrum envelope, to obtain coefficients in the logarithm domain;

respectively performing a gain adjustment on each of the coefficients in the logarithm domain based on the difference value corresponding to the to-be-processed subband spectrum envelope, to obtain adjusted coefficients; and

respectively performing an inverse logarithmic transformation on each of the adjusted coefficients, to obtain adjusted spectrum coefficients corresponding to the to-be-processed subband spectrum envelope.

7. The method according to claim 1 , wherein the obtaining the speech feature parameters comprises:

performing a time-frequency transform on the to-be-processed speech signal to obtain the initial spectrum, wherein

when the time-frequency transform is a Fourier transform (FT),

the initial spectrum is an initial amplitude spectrum,

the predicted subband spectrum envelopes are subband amplitude spectrum envelopes,

the adjusting the initial spectrum based on the predicted subband spectrum envelopes to obtain the adjusted spectrum includes

adjusting the initial amplitude spectrum based on the predicted subband amplitude spectrum envelopes to obtain an adjusted amplitude spectrum, and

the obtaining the enhanced speech signal includes

obtaining a target phase spectrum based on an initial phase spectrum of the to-be-processed speech signal, and

performing a time-frequency inverse transform based on the adjusted amplitude spectrum and the target phase spectrum to obtain the enhanced speech signal; and

when the time-frequency transform is a discrete cosine transform (DCT),

the obtaining the enhanced speech signal includes

performing the time-frequency inverse transform on the adjusted spectrum, to obtain the enhanced speech signal.

8. The method according to claim 1 , wherein the predicting comprises:

predicting the first quantity of subband spectrum envelopes based on the speech feature parameters by using a neural network,

wherein the neural network is obtained through a training process that includes:

obtaining training data, the training data comprising multiple to-be-enhanced sample speech signals and multiple pure speech signals, each pure speech signal corresponding to a different to-be-enhanced sample speech signal;

obtaining sample subband spectrum envelopes for each to-be-enhanced sample speech signal based on the pure speech signal corresponding to the respective to-be-enhanced sample speech signal;

determining speech feature parameters of each to-be-enhanced sample speech signal; and

training an initial neural network based on each to-be-enhanced sample speech signal until a loss function of the initial neural network converges, an input of the initial neural network being the speech feature parameters of each to-be-enhanced sample speech signal, an output of the initial neural network being predicted subband spectrum envelopes corresponding to each to-be-enhanced sample speech signal, and a value of the loss function representing a difference between the predicted subband spectrum envelopes corresponding to each to-be-enhanced sample speech signal and the corresponding sample subband spectrum envelopes.

9. The method according to claim 8 , wherein for one pure speech signal, the obtaining the sample subband spectrum envelopes comprises:

performing a time-frequency transform on the pure speech signal to obtain an initial spectrum of the pure speech signal;

dividing the initial spectrum of the pure speech signal into a first quantity of spectrum subbands, each of the spectrum subbands including a second quantity of spectrum coefficients in the initial spectrum; and

obtaining a respective sample subband spectrum envelope for each spectrum subband based on the spectrum coefficients included in the respective spectrum subband to obtain the sample subband spectrum envelopes for the respective to-be-enhanced sample speech signal corresponding to the pure speech signal.

10. The method according to claim 8 , wherein the neural network comprises an input layer, a hidden layer, and an output layer that are sequentially cascaded,

the hidden layer comprising a long short-term memory (LSTM) layer and a fully-connected layer that are sequentially cascaded,

an input of the input layer being a first feature vector of the speech feature parameters,

the LSTM layer being configured to perform a feature extraction on the first feature vector processed by the input layer to generate a second feature vector,

the fully-connected layer being configured to perform the feature extraction on the second feature vector generated by the LSTM layer to generate a third feature vector, and

the output layer being configured to obtain an output vector with a first quantity of dimensions based on the third feature vector generated by the fully-connected layer, element values of elements in the output vector being the predicted subband spectrum envelopes.

11. A speech signal processing apparatus, comprising:

processing circuitry configured to:

obtain speech feature parameters of a to-be-processed speech signal, the speech feature parameters including an initial spectrum of the to-be-processed speech signal;

predict a first quantity of subband spectrum envelopes based on the speech feature parameters, the first quantity being less than a quantity of initial spectrum coefficients included in the initial spectrum, and each of the subband spectrum envelopes corresponding to a second quantity of initial spectrum coefficients;

divide the initial spectrum into a first quantity of spectrum subbands, each of the spectrum subbands including the second quantity of initial spectrum coefficients in the initial spectrum;

obtain a respective to-be-processed subband spectrum envelope for each spectrum subband based on the initial spectrum coefficients included in the respective spectrum subband;

determine a respective difference value for each of the predicted subband spectrum envelopes between the respective predicted subband spectrum envelope and a corresponding to-be-processed subband spectrum envelope;

adjust the initial spectrum coefficients for each to-be-processed subband spectrum envelope, based on the difference value corresponding to the respective to-be-processed subband spectrum envelope, to obtain an adjusted spectrum; and

obtain an enhanced speech signal corresponding to the to-be-processed speech signal based on the adjusted spectrum.

12. The apparatus according to claim 11 , wherein the speech feature parameters further include time domain feature parameters of the to-be-processed speech signal.

13. The apparatus according to claim 11 , wherein both the predicted subband spectrum envelopes and the to-be-processed subband spectrum envelopes are in a logarithm domain, and for one to-be-processed subband spectrum envelope, the processing circuitry is configured to:

respectively perform a logarithmic transformation on each of the initial spectrum coefficients corresponding to the to-be-processed subband spectrum envelope, to obtain coefficients in the logarithm domain;

respectively perform a gain adjustment on each of the coefficients in the logarithm domain based on the difference value corresponding to the to-be-processed subband spectrum envelope, to obtain adjusted coefficients; and

respectively perform an inverse logarithmic transformation on each of the adjusted coefficients, to obtain adjusted spectrum coefficients corresponding to the to-be-processed subband spectrum envelope.

14. The apparatus according to claim 11 , wherein the processing circuitry is configured to:

perform a time-frequency transform on the to-be-processed speech signal to obtain the initial spectrum, wherein

when the time-frequency transform is a Fourier transform (FT),

the initial spectrum is an initial amplitude spectrum,

the predicted subband spectrum envelopes are subband amplitude spectrum envelopes,

the processing circuitry is configured to

adjust the initial amplitude spectrum based on the predicted subband amplitude spectrum envelopes to obtain an adjusted amplitude spectrum,

obtain a target phase spectrum based on an initial phase spectrum of the to-be-processed speech signal, and

perform a time-frequency inverse transform based on the adjusted amplitude spectrum and the target phase spectrum to obtain the enhanced speech signal; and

when the time-frequency transform is a discrete cosine transform (DCT), a time-frequency inverse transform is performed on the adjusted spectrum, to obtain the enhanced speech signal.

15. The apparatus according to claim 11 , wherein the processing circuitry is configured to:

predict the first quantity of subband spectrum envelopes based on the speech feature parameters by using a neural network,

wherein the neural network is obtained through a training process that includes:

obtaining training data, the training data comprising multiple to-be-enhanced sample speech signals and multiple pure speech signals, each pure speech signal corresponding to a different to-be-enhanced sample speech signal;

obtaining sample subband spectrum envelopes for each to-be-enhanced sample speech signal based on the pure speech signal corresponding to the respective to-be-enhanced sample speech signal;

determining speech feature parameters of each to-be-enhanced sample speech signal; and

training an initial neural network based on each to-be-enhanced sample speech signal until a loss function of the initial neural network converges, an input of the initial neural network being the speech feature parameters of each to-be-enhanced sample speech signal, an output of the initial neural network being predicted subband spectrum envelopes corresponding to each to-be-enhanced sample speech signal, and a value of the loss function representing a difference between the predicted subband spectrum envelopes corresponding to each to-be-enhanced sample speech signal and the corresponding sample subband spectrum envelopes.

16. The apparatus according to claim 15 , wherein for one pure speech signal, the obtaining the sample subband spectrum envelopes comprises:

performing a time-frequency transform on the pure speech signal to obtain an initial spectrum of the pure speech signal;

dividing the initial spectrum of the pure speech signal into a first quantity of spectrum subbands, each of the spectrum subbands including a second set quantity of spectrum coefficients in the initial spectrum; and

obtaining a respective sample subband spectrum envelope for each spectrum subband based on the spectrum coefficients included in the respective spectrum subband to obtain the sample subband spectrum envelopes for the respective to-be-enhanced sample speech signal corresponding to the pure speech signal.

17. The apparatus according to claim 15 , wherein the neural network comprises an input layer, a hidden layer, and an output layer that are sequentially cascaded,

the hidden layer comprising a long short-term memory (LSTM) layer and a fully-connected layer that are sequentially cascaded,

an input of the input layer being a first feature vector of the speech feature parameters,

the LSTM layer being configured to perform a feature extraction on the first feature vector processed by the input layer to generate a second feature vector,

the fully-connected layer being configured to perform the feature extraction on the second feature vector generated by the LSTM layer to generate a third feature vector, and

the output layer being configured to obtain an output vector with a first quantity of dimensions based on the third feature vector generated by the fully-connected layer, element values of elements in the output vector being the predicted subband spectrum envelopes.

18. A non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor cause the at least one processor to perform:

obtaining speech feature parameters of a to-be-processed speech signal, the speech feature parameters including an initial spectrum of the to-be-processed speech signal;

predicting a first quantity of subband spectrum envelopes based on the speech feature parameters, the first quantity being less than a quantity of initial spectrum coefficients included in the initial spectrum, and each of the subband spectrum envelopes corresponding to a second quantity of initial spectrum coefficients;

dividing the initial spectrum into a first quantity of spectrum subbands, each of the spectrum subbands including the second quantity of initial spectrum coefficients in the initial spectrum;

obtaining a respective to-be-processed subband spectrum envelope for each spectrum subband based on the initial spectrum coefficients included in the respective spectrum subband;

determining a respective difference value for each of the predicted subband spectrum envelopes between the respective predicted subband spectrum envelope and a corresponding to-be-processed subband spectrum envelope;

adjusting the initial spectrum coefficients for each to-be-processed subband spectrum envelope, based on the difference value corresponding to the respective to-be-processed subband spectrum envelope, to obtain an adjusted spectrum; and

obtaining an enhanced speech signal corresponding to the to-be-processed speech signal based on the adjusted spectrum.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (9)