Speech processing method and terminal
A speech processing method and a terminal are provided. The method includes: receiving signals from a plurality of microphones; performing, by using a same sampling rate, analog-to-digital conversion on the plurality of paths of signals received from the plurality of microphones, to obtain a plurality of paths of time-domain digital signals; performing time-to-frequency-domain conversion on the plurality of paths of time-domain digital signals to obtain a plurality of paths of frequency-domain signals; and determining a signal type of the primary frequency-domain signal based on at least one of a sound pressure difference between the primary frequency-domain signal and each of N paths of secondary frequency-domain signals in the M paths of secondary frequency-domain signals, a phase difference between the primary frequency-domain signal and each of the N paths of secondary frequency-domain signals, and a frequency distribution characteristic of the primary frequency-domain signal.
1. A speech processing method, comprising:
receiving, by a terminal, signals from a plurality of microphones, wherein the plurality of microphones includes a primary microphone and M secondary microphones, wherein M≥1, and M is an integer;
performing, by the terminal, analog-to-digital conversion on the signals received from the plurality of microphones to obtain a plurality of time-domain digital signals, wherein a same sampling rate is used to perform the analog-to-digital conversion for each of the signals received from the plurality of microphones;
performing, by the terminal, time-to-frequency-domain conversion on the plurality of time-domain digital signals to obtain a plurality of frequency-domain signals, wherein the plurality of frequency-domain signals includes a primary frequency-domain signal and M secondary frequency-domain signals, wherein the primary frequency-domain signal corresponds to the primary microphone, and the M secondary frequency-domain signals are in a one-to-one correspondence with the M secondary microphones, respectively; and
determining, by the terminal, a signal type of the primary frequency-domain signal based on at least one of: a sound pressure difference between the primary frequency-domain signal and each of N secondary frequency-domain signals in the M secondary frequency-domain signals, a phase difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals, and a frequency distribution characteristic of the primary frequency-domain signal, wherein 1≤N≤M, and N is an integer;
wherein determining the signal type of the primary frequency-domain signal comprises determining the primary frequency-domain signal as a whisper signal in response to determining that each of the following conditions is satisfied:
the sound pressure difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals is greater than or equal to a corresponding seventh threshold, wherein each of the N secondary frequency-domain signals corresponds to one seventh threshold;
the phase difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals is greater than or equal to a corresponding eighth threshold, wherein each of the N secondary frequency-domain signals corresponds to one eighth threshold; and
the primary frequency-domain signal satisfies at least one of the following conditions: there is no fundamental frequency; a harmonic quantity is less than or equal to a fifth threshold; and a sub-band having maximum sub-band energy belongs to a first frequency band, and a difference between normalized energy of each of at least three sub-bands comprised in the primary frequency-domain signal and normalized energy of a corresponding sub-band in at least three sub-bands comprised in a predefined whisper signal is less than a corresponding sixth threshold, wherein the at least three sub-bands comprised in the primary frequency-domain signal are in a one-to-one correspondence with the at least three sub-bands comprised in the predefined whisper signal, each of the at least three sub-bands comprised in the primary frequency-domain signal corresponds to a corresponding sixth threshold, the normalized energy of each sub-band is a ratio of energy of the sub-band to total energy of the at least three sub-bands, and the energy of the sub-band is a sum of energy of frequency ranges in the sub-band.
2. The method according to claim 1 , wherein in response to determining that the primary frequency-domain signal is a whisper signal, the method further comprises:
calculating an estimated noise based on performing at least one of: stationary noise estimation, background human noise estimation, and burst noise estimation on the primary frequency-domain signal; and
performing noise reduction processing on the estimated noise.
3. The method according to claim 2 , wherein the performing background human noise estimation on the primary frequency-domain signal comprises:
performing background human noise estimation on the primary frequency-domain signal by using an adaptive beamformer whose beam width is a first width, wherein the first width is less than a preset beam width.
4. The method according to claim 3 , wherein the performing burst noise estimation on the primary frequency-domain signal comprises:
collecting statistics about a change speed of the primary frequency-domain signal in a first time window by using the first time window whose time window length is less than a preset length;
in response to determining that the change speed of the primary frequency-domain signal is greater than a preset speed threshold, determining whether normalized energy of each sub-band of the primary frequency-domain signal in the first time window is greater than or equal to a corresponding ninth threshold, and whether normalized energy of all sub-bands of the primary frequency-domain signal in the first time window is diminished in ascending order of frequencies, wherein the primary frequency-domain signal in the first time window comprises at least one sub-band, the normalized energy of each sub-band is a ratio of energy of the sub-band to total energy of the at least one sub-band, and the energy of the sub-band is a sum of energy of frequency ranges in the sub-band; and
determining the signal in the first time window as burst noise in response to determining that the normalized energy of each sub-band of the primary frequency-domain signal in the first time window is greater than or equal to the corresponding ninth threshold and the normalized energy of all the sub-bands of the primary frequency-domain signal in the first time window is diminished in ascending order of frequencies.
5. The method according to claim 3 , wherein the method further comprises:
performing gain control on a signal obtained after the noise reduction, to obtain a gain-adjusted signal.
6. The method according to claim 5 , wherein in response to determining that gain control is performed on the signal obtained after the noise reduction, at least one of the following conditions is satisfied:
a gain lower limit is a target lower limit,
a gain upper limit is a target upper limit,
a maximum level value of the signal obtained after the noise reduction is a target level value, and
a gain smoothing speed is a target speed,
wherein the target lower limit is greater than a preset lower limit, the target upper limit is less than a preset upper limit, the target level value is less than a preset level value, and the target speed is less than a preset speed.
7. The method according to claim 5 , wherein the method further comprises:
performing frequency response control on the gain-adjusted signal, to obtain a frequency-response-adjusted signal.
8. The method according to claim 7 , wherein the performing frequency response control on the gain-adjusted signal comprises:
performing frequency response control on a first frequency band of the gain-adjusted signal by using a first target frequency response,
performing frequency response control on a second frequency band of the gain-adjusted signal by using a second target frequency response, and
performing frequency response control on a third frequency band of the gain-adjusted signal by using a third target frequency response,
wherein the first target frequency response is greater than or equal to a first preset frequency response, the second target frequency response is greater than a second preset frequency response, the third target frequency response is less than a third preset frequency response, and a minimum frequency in the first frequency band is greater than or equal to a maximum frequency in the second frequency band and is less than or equal to a minimum frequency in the third frequency band.
9. A terminal, comprising:
a receiver, configured to receive signals from a plurality of microphones, wherein the plurality of microphones includes a primary microphone and M secondary microphones, wherein M≥1, and M is an integer; and
a processor configured to:
perform analog-to-digital conversion on the plurality of signals received from the plurality of microphones to obtain a plurality of time-domain digital signals, wherein a same sampling rate is used to perform the analog-to-digital conversion for each of the signals received from the plurality of microphones;
perform time-to-frequency-domain conversion on the plurality of time-domain digital signals to obtain a plurality of frequency-domain signals, wherein the plurality of frequency-domain signals includes a primary frequency-domain signal and M secondary frequency-domain signals, wherein the primary frequency-domain signal corresponds to the primary microphone, and the M secondary frequency-domain signals are in a one-to-one correspondence with the M secondary microphones, respectively; and
determine a signal type of the primary frequency-domain signal based on at least one of a sound pressure difference between the primary frequency-domain signal and each of N secondary frequency-domain signals in the M secondary frequency-domain signals, a phase difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals, and a frequency distribution characteristic of the primary frequency-domain signal, wherein 1≤N≤M, and N is an integer;
wherein determining the signal type of the primary frequency-domain signal comprises determining the primary frequency-domain signal as a whisper signal in response to determining that each of the following conditions is satisfied:
the sound pressure difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals is greater than or equal to a corresponding seventh threshold, wherein each of the N secondary frequency-domain signals corresponds to one seventh threshold;
the phase difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals is greater than or equal to a corresponding eighth threshold, wherein each of the N secondary frequency-domain signals corresponds to one eighth threshold; and
the primary frequency-domain signal satisfies at least one of the following conditions: there is no fundamental frequency; a harmonic quantity is less than or equal to a fifth threshold; and a sub-band having maximum sub-band energy belongs to a first frequency band, and a difference between normalized energy of each of at least three sub-bands comprised in the primary frequency-domain signal and normalized energy of a corresponding sub-band in at least three sub-bands comprised in a predefined whisper signal is less than a corresponding sixth threshold, wherein the at least three sub-bands comprised in the primary frequency-domain signal are in a one-to-one correspondence with the at least three sub-bands comprised in the predefined whisper signal, each of the at least three sub-bands comprised in the primary frequency-domain signal corresponds to a corresponding sixth threshold, the normalized energy of each sub-band is a ratio of energy of the sub-band to total energy of the at least three sub-bands, and the energy of the sub-band is a sum of energy of frequency ranges in the sub-band.
10. The terminal according to claim 9 , wherein the processor is further configured to:
calculate an estimated noise based on performing at least one of stationary noise estimation, background human noise estimation, and burst noise estimation on the primary frequency-domain signal; and
perform noise reduction processing on the estimated noise.
11. The terminal according to claim 10 , wherein the processor is further configured to:
perform background human noise estimation on the primary frequency-domain signal by using an adaptive beamformer whose beam width is a first width, wherein the first width is less than a preset beam width.
12. The terminal according to claim 11 , wherein the processor is further configured to:
collect statistics about a change speed of the primary frequency-domain signal in a first time window by using the first time window whose time window length is less than a preset length;
in response to determining that the change speed of the signal is greater than a preset speed threshold, determine whether normalized energy of each sub-band of the primary frequency-domain signal in the first time window is greater than or equal to a corresponding ninth threshold, and whether normalized energy of all sub-bands of the primary frequency-domain signal in the first time window is diminished in ascending order of frequencies, wherein the primary frequency-domain signal in the first time window comprises at least one sub-band, the normalized energy of each sub-band is a ratio of energy of the sub-band to total energy of the at least one sub-band, and the energy of the sub-band is a sum of energy of frequency ranges in the sub-band; and
determine the signal in the first time window as burst noise in response to determining that the normalized energy of each sub-band of the primary frequency-domain signal in the first time window is greater than or equal to the corresponding ninth threshold and the normalized energy of all the sub-bands of the primary frequency-domain signal in the first time window is diminished in ascending order of frequencies.
13. The terminal according to claim 11 , wherein the processor is further configured to:
perform gain control on a signal obtained after the noise reduction, to obtain a gain-adjusted signal.
14. The terminal according to claim 13 , wherein in response to determining that gain control is performed on the signal obtained after the noise reduction, at least one of the following conditions is satisfied:
a gain lower limit is a target lower limit, a gain upper limit is a target upper limit, a maximum level value of the signal obtained after the noise reduction is a target level value, and a gain smoothing speed is a target speed,
wherein the target lower limit is greater than a preset lower limit, the target upper limit is less than a preset upper limit, the target level value is less than a preset level value, and the target speed is less than a preset speed.
15. The terminal according to claim 13 , wherein the processor is further configured to:
perform frequency response control on the gain-adjusted signal, to obtain a frequency-response-adjusted signal.
16. The terminal according to claim 15 , wherein the processor is further configured to:
perform frequency response control on a first frequency band of the gain-adjusted signal by using a first target frequency response,
perform frequency response control on a second frequency band of the gain-adjusted signal by using a second target frequency response, and
perform frequency response control on a third frequency band of the gain-adjusted signal by using a third target frequency response,
wherein the first target frequency response is greater than or equal to a first preset frequency response, the second target frequency response is greater than a second preset frequency response, the third target frequency response is less than a third preset frequency response, and a minimum frequency in the first frequency band is greater than or equal to a maximum frequency in the second frequency band and is less than or equal to a minimum frequency in the third frequency band.
17. A non-transitory computer-readable storage medium storing instructions that, when executed, cause a terminal to perform the steps of:
receiving signals from a plurality of microphones, wherein the plurality of microphones includes a primary microphone and M secondary microphones, wherein M≥1, and M is an integer;
performing analog-to-digital conversion on the plurality of signals received from the plurality of microphones to obtain a plurality of time-domain digital signals, wherein a same sampling rate is used to perform the analog-to-digital conversion for each of the signals received from the plurality of microphones;
performing time-to-frequency-domain conversion on the plurality of time-domain digital signals to obtain a plurality of frequency-domain signals, wherein the plurality of frequency-domain signals includes a primary frequency-domain signal and M secondary frequency-domain signals, wherein the primary frequency-domain signal corresponds to the primary microphone, and the M secondary frequency-domain signals are in a one-to-one correspondence with the M secondary microphones, respectively; and
determining a signal type of the primary frequency-domain signal based on: a sound pressure difference between the primary frequency-domain signal and each of N secondary frequency-domain signals in the M secondary frequency-domain signals and a frequency distribution characteristic of the primary frequency-domain signal, wherein 1≤N≤M, and N is an integer;
wherein determining the signal type of the primary frequency-domain signal comprises determining the primary frequency-domain signal as a whisper signal in response to determining that each of the following conditions is satisfied:
the sound pressure difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals is greater than or equal to a corresponding seventh threshold, wherein each of the N secondary frequency-domain signals corresponds to one seventh threshold;
the phase difference between the primary frequency-domain signal and each of the N secondary frequency-domain signals is greater than or equal to a corresponding eighth threshold, wherein each of the N secondary frequency-domain signals corresponds to one eighth threshold; and
the primary frequency-domain signal satisfies at least one of the following conditions: there is no fundamental frequency; a harmonic quantity is less than or equal to a fifth threshold; and a sub-band having maximum sub-band energy belongs to a first frequency band, and a difference between normalized energy of each of at least three sub-bands comprised in the primary frequency-domain signal and normalized energy of a corresponding sub-band in at least three sub-bands comprised in a predefined whisper signal is less than a corresponding sixth threshold,
wherein the at least three sub-bands comprised in the primary frequency-domain signal are in a one-to-one correspondence with the at least three sub-bands comprised in the predefined whisper signal, each of the at least three sub-bands comprised in the primary frequency-domain signal corresponds to a corresponding sixth threshold, the normalized energy of each sub-band is a ratio of energy of the sub-band to total energy of the at least three sub-bands, and the energy of the sub-band is a sum of energy of frequency ranges in the sub-band.
18. The computer-readable storage medium according to claim 17 , wherein in response to determining that the primary frequency-domain signal is a whisper signal, executing the instructions further causes the terminal to perform the steps of:
calculating an estimated noise based on performing at least one of: stationary noise estimation, background human noise estimation, and burst noise estimation on the primary frequency-domain signal; and
performing noise reduction processing on the estimated noise.
19. The computer-readable storage medium according to claim 18 , wherein calculating the estimated noise comprises:
performing background human noise estimation on the primary frequency-domain signal by using an adaptive beamformer whose beam width is a first width, wherein the first width is less than a preset beam width.
20. The computer-readable storage medium according to claim 19 , wherein calculating the estimated noise comprises:
collecting statistics about a change speed of the primary frequency-domain signal in a first time window by using the first time window whose time window length is less than a preset length;
in response to determining that the change speed of the signal is greater than a preset speed threshold, determining whether normalized energy of each sub-band of the primary frequency-domain signal in the first time window is greater than or equal to a corresponding ninth threshold, and whether normalized energy of all sub-bands of the primary frequency-domain signal in the first time window is diminished in ascending order of frequencies, wherein the primary frequency-domain signal in the first time window comprises at least one sub-band, the normalized energy of each sub-band is a ratio of energy of the sub-band to total energy of the at least one sub-band, and the energy of the sub-band is a sum of energy of frequency ranges in the sub-band; and
determining the signal in the first time window as burst noise in response to determining that the normalized energy of each sub-band of the primary frequency-domain signal in the first time window is greater than or equal to the corresponding ninth threshold and the normalized energy of all the sub-bands of the primary frequency-domain signal in the first time window is diminished in ascending order of frequencies.
21. The computer-readable storage medium according to claim 19 , wherein executing the instructions further causes the terminal to perform the steps of:
performing gain control on a signal obtained after the noise reduction, to obtain a gain-adjusted signal.
22. The computer-readable storage medium according to claim 21 , wherein in response to determining that gain control is performed on the signal obtained after the noise reduction, at least one of the following conditions is satisfied:
a gain lower limit is a target lower limit, a gain upper limit is a target upper limit, a maximum level value of the signal obtained after the noise reduction is a target level value, and a gain smoothing speed is a target speed,
wherein the target lower limit is greater than a preset lower limit, the target upper limit is less than a preset upper limit, the target level value is less than a preset level value, and the target speed is less than a preset speed.
23. The computer-readable storage medium according to claim 21 , wherein executing the instructions further causes the terminal to perform the steps of:
performing frequency response control on the gain-adjusted signal, to obtain a frequency-response-adjusted signal.
24. The computer-readable storage medium according to claim 23 , wherein performing frequency response control comprises:
performing frequency response control on a first frequency band of the gain-adjusted signal by using a first target frequency response,
performing frequency response control on a second frequency band of the gain-adjusted signal by using a second target frequency response, and
performing frequency response control on a third frequency band of the gain-adjusted signal by using a third target frequency response,
wherein the first target frequency response is greater than or equal to a first preset frequency response, the second target frequency response is greater than a second preset frequency response, the third target frequency response is less than a third preset frequency response, and a minimum frequency in the first frequency band is greater than or equal to a maximum frequency in the second frequency band and is less than or equal to a minimum frequency in the third frequency band.