IP Library › Granted Patent US 11,848,029
Granted Patent B2
US 11,848,029 · App. 17/327,224 · Granted Dec 19, 2023

Method and device for detecting audio signal, and storage medium

Inventors: Yifeng Wang (Beijing, CN); Guodu Cai (Beijing, CN); Shuo Yang (Beijing, CN); Lihan Li (Beijing, CN); Peng Gao (Beijing, CN)
Assignee: BEIJING XIAOMI PINECONE ELECTRONICS CO., LTD.
G10L25/87G10L15/02G10L15/04G10L15/063G10L15/142G10L15/22G10L25/18G10L25/21G10L25/24G10L25/93G10L2015/025
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,848,029
App. No.
17/327,224
Granted
Dec 19, 2023
Kind
B2
Abstract

A method for detecting an audio signal, the method comprises: obtaining a speech segment and a non-speech segment of an audio signal to be detected, extracting a first audio feature of the speech segment and a second audio feature of the non-speech segment, detecting the first audio feature using a predetermined speech segment detection model to obtain a first detection score, detecting the second audio feature using a predetermined non-speech segment detection model to obtain a second detection score, and determining whether the audio signal belongs to a target audio based on the first detection score and the second detection score.

Claims (98)

1. A method for detecting an audio signal, comprising:

obtaining a speech segment and a non-speech segment of an audio signal to be detected;

extracting a first audio feature of the speech segment and a second audio feature of the non-speech segment;

detecting the first audio feature using a predetermined speech segment detection model to obtain a first detection score;

detecting the second audio feature using a predetermined non-speech segment detection model to obtain a second detection score; and

determining whether the audio signal belongs to a target audio based on the first detection score and the second detection score;

wherein the extracting the first audio feature of the speech segment comprises:

determining an extraction frequency band on which the first audio feature is extracted based on a relevancy between the audio signal and a predetermined speech content that is textual information existing in a lower frequency band; and

extracting the first audio feature on the extraction frequency band from the speech segment;

wherein determining the extraction frequency band of the first audio feature based on the relevancy between the audio signal and the predetermined speech content comprises:

in response to the relevancy being less than a predetermined threshold, determining a predetermined first frequency band as the extraction frequency band;

in response to the relevancy being greater than or equal to the predetermined threshold, determining a predetermined second frequency band as the extraction frequency band, wherein the predetermined second frequency band has a bandwidth larger than that of the predetermined first frequency band and contains the predetermined first frequency band.

2. The method of claim 1 , wherein the obtaining the speech segment and the non-speech segment of the audio signal to be detected comprises:

recognizing a first silent segment in the audio signal using a first recognition way;

recognizing an unvoiced sound segment and a second silent segment in the audio signal using a second recognition way;

determining a union of the unvoiced sound segment, the first silent segment and the second silent segment as the non-speech segment; and

determining an audio segment other than the non-speech segment in the audio signal as the speech segment.

3. The method of claim 2 , wherein the recognizing the first silent segment in the audio signal using the first recognition way comprises:

performing an energy detection on the audio signal with a voice activity detection (VAD); and

determining an audio segment having energy less than a predetermined threshold in the audio signal as the first silent segment.

4. The method of claim 2 , wherein the recognizing the unvoiced sound segment and the second silent segment in the audio signal using the second recognition way comprises:

generating a phoneme set of the audio signal through a hidden markov model (HMM);

performing a phoneme alignment on the phoneme set with a montreal forced aligner (MFA); and

obtaining the unvoiced sound segment and the second silent segment in the phoneme set after the phoneme alignment.

5. The method of claim 1 , wherein, the predetermined speech segment detection model includes a first real sound model and a first attack sound model, the detecting the first audio feature using the predetermined speech segment detection model to obtain the first detection score comprises:

detecting the first audio feature using the first real sound model to obtain a first real sound score;

detecting the first audio feature using the first attack sound model to obtain a first attack sound score; and

determining the first detection score based on the first real sound score and the first attack sound score.

6. The method of claim 1 , wherein, the predetermined non-speech segment detection model includes a second real sound model and a second attack sound model, the detecting the second audio feature using the predetermined non-speech segment detection model to obtain the second detection score comprises:

detecting the second audio feature using the second real sound model to obtain a second real sound score;

detecting the second audio feature using the second attack sound model to obtain a second attack sound score; and

determining the second detection score based on the second real sound score and the second attack sound score.

7. The method of claim 1 , wherein the extracting the first audio feature of the speech segment and the second audio feature of the non-speech segment comprises:

obtaining a number of collecting channels of the audio signal; and

extracting the first audio feature of the speech segment and the second audio feature of the non-speech segment with a cepstral mean and variance normalization (CMVN) when the number of collecting channels satisfies a predetermined number condition.

8. The method of claim 1 , further comprising:

performing a training based on speech data in predetermined audio data to obtain the speech segment detection model; and

performing a training based on non-speech data in the predetermined audio data to obtain the non-speech segment detection model.

9. The method of claim 8 , wherein, the predetermined audio data includes predetermined real sound data and predetermined attack sound data, the predetermined speech segment detection model includes a first real sound model and a first attack sound model, the predetermined non-speech segment detection model includes a second real sound model and a second attack sound model;

wherein performing the training based on the speech data in the predetermined audio data to obtain the speech segment detection model comprises:

performing the training based on the speech data in the predetermined real sound data to obtain the first real sound model; and

performing the training based on the speech data in the predetermined attack sound data to obtain the first attack sound model;

wherein performing the training based on the non-speech data in the predetermined audio data to obtain the non-speech segment detection model comprises:

performing the training based on the non-speech data in the predetermined real sound data to obtain the second real sound model; and

performing the training based on the non-speech data in the predetermined attack sound data to obtain the second attack sound model.

10. A device for detecting an audio signal, comprising:

a processor; and

a memory for storing executable instructions capable of running on the processor;

wherein when the processor is configured to execute the executable instructions, the executable instructions perform the following:

obtaining a speech segment and a non-speech segment of an audio signal to be detected;

extracting a first audio feature of the speech segment and a second audio feature of the non-speech segment;

detecting the first audio feature using a predetermined speech segment detection model to obtain a first detection score;

detecting the second audio feature using a predetermined non-speech segment detection model to obtain a second detection score; and

determining whether the audio signal belongs to a target audio based on the first detection score and the second detection score;

wherein the extracting the first audio feature of the speech segment comprises:

determining an extraction frequency band on which the first audio feature is extracted based on a relevancy between the audio signal and a predetermined speech content that is textual information existing in a lower frequency band; and

extracting the first audio feature on the extraction frequency band from the speech segment;

wherein determining the extraction frequency band of the first audio feature based on the relevancy between the audio signal and the predetermined speech content comprises:

in response to the relevancy being less than a predetermined threshold, determining a predetermined first frequency band as the extraction frequency band;

in response to the relevancy being greater than or equal to the predetermined threshold, determining a predetermined second frequency band as the extraction frequency band, wherein the predetermined second frequency band has a bandwidth larger than that of the predetermined first frequency band and contains the predetermined first frequency band.

11. The device of claim 10 , wherein the obtaining the speech segment and the non-speech segment of the audio signal to be detected comprises:

recognizing a first silent segment in the audio signal using a first recognition way;

recognizing an unvoiced sound segment and a second silent segment in the audio signal using a second recognition way;

determining a union of the unvoiced sound segment, the first silent segment and the second silent segment as the non-speech segment; and

determining an audio segment other than the non-speech segment in the audio signal as the speech segment.

12. The device of claim 11 , wherein the recognizing the first silent segment in the audio signal using the first recognition way comprises:

performing an energy detection on the audio signal with a voice activity detection (VAD); and

determining an audio segment having energy less than a predetermined threshold in the audio signal as the first silent segment.

13. The device of claim 11 , wherein the recognizing the unvoiced sound segment and the second silent segment in the audio signal using the second recognition way comprises:

generating a phoneme set of the audio signal through a hidden markov model (HMM);

performing a phoneme alignment on the phoneme set with a montreal forced aligner (MFA); and

obtaining the unvoiced sound segment and the second silent segment in the phoneme set after the phoneme alignment.

14. The device of claim 10 , wherein, the predetermined speech segment detection model includes a first real sound model and a first attack sound model, the detecting the first audio feature using the predetermined speech segment detection model to obtain the first detection score comprises:

detecting the first audio feature using the first real sound model to obtain a first real sound score;

detecting the first audio feature using the first attack sound model to obtain a first attack sound score; and

determining the first detection score based on the first real sound score and the first attack sound score.

15. The device of claim 10 , wherein, the predetermined non-speech segment detection model includes a second real sound model and a second attack sound model, the detecting the second audio feature using the predetermined non-speech segment detection model to obtain the second detection score comprises:

detecting the second audio feature using the second real sound model to obtain a second real sound score;

detecting the second audio feature using the second attack sound model to obtain a second attack sound score; and

determining the second detection score based on the second real sound score and the second attack sound score.

16. The device of claim 10 , wherein the extracting the first audio feature of the speech segment and the second audio feature of the non-speech segment comprises:

obtaining a number of collecting channels of the audio signal; and

extracting the first audio feature of the speech segment and the second audio feature of the non-speech segment with a cepstral mean and variance normalization (CMVN) when the number of collecting channels satisfies a predetermined number condition.

17. The device of claim 10 , wherein the executable instructions further perform the following:

performing a training based on speech data in predetermined audio data to obtain the speech segment detection model; and

performing a training based on non-speech data in the predetermined audio data to obtain the non-speech segment detection model.

18. A non-transitory computer readable storage medium having computer executable instructions stored therein, wherein when the computer executable instructions are executed by a processor, the processor is caused to perform the following:

obtaining a speech segment and a non-speech segment of an audio signal to be detected;

extracting a first audio feature of the speech segment and a second audio feature of the non-speech segment;

detecting the first audio feature using a predetermined speech segment detection model to obtain a first detection score;

detecting the second audio feature using a predetermined non-speech segment detection model to obtain a second detection score; and

determining whether the audio signal belongs to a target audio based on the first detection score and the second detection score;

wherein the extracting the first audio feature of the speech segment comprises:

determining an extraction frequency band on which the first audio feature is extracted based on a relevancy between the audio signal and a predetermined speech content that is textual information existing in a lower frequency band; and

extracting the first audio feature on the extraction frequency band from the speech segment;

wherein determining the extraction frequency band of the first audio feature based on the relevancy between the audio signal and the predetermined speech content comprises:

in response to the relevancy being less than a predetermined threshold, determining a predetermined first frequency band as the extraction frequency band;

in response to the relevancy being greater than or equal to the predetermined threshold, determining a predetermined second frequency band as the extraction frequency band, wherein the predetermined second frequency band has a bandwidth larger than that of the predetermined first frequency band and contains the predetermined first frequency band.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2021
From: WANG, YIFENG; CAI, GUODU; YANG, SHUO; LI, LIHAN; GAO, PENG
To: BEIJING XIAOMI PINECONE ELECTRONICS CO., LTD.
Reel/Frame 056317/0607 →
Priority Claims (1)
CN 202011312515.7 · Nov 20, 2020 · national
Continuity (1)
Related Publication 20220165297A1 · May 26, 2022