Method for detecting and classifying coughs or other non-semantic sounds using audio feature set learned from speech
A method of detecting a cough in an audio stream includes a step of performing one or more pre-processing steps on the audio stream to generate an input audio sequence comprising a plurality of time-separated audio segments. An embedding is generated by a self-supervised triplet loss embedding model for each of the segments of the input audio sequence using an audio feature set, the embedding model having been trained to learn the audio feature set in a self-supervised triplet loss manner from a plurality of speech audio clips from a speech dataset. The embedding for each of the segments is provided to a model performing cough detection inference. This model generates a probability that each of the segments of the input audio sequence includes a cough episode. The method includes generating cough metrics for each of the cough episodes detected in the input audio sequence.
1. A computer-implemented method of detecting a non-semantic and paralinguistic event in an audio stream comprising:
performing one or more pre-processing steps on the audio stream to generate an input audio sequence comprising a plurality of time-separated audio segments;
generating, by a student model, an embedding for the plurality of time-separated audio segments, the student model having been trained using knowledge distillation applied to a self-supervised triplet loss embedding model, the self-supervised triplet loss embedding model having been trained to learn an audio feature set in a self-supervised triplet loss manner from a plurality of speech audio clips; and
providing the embedding for the plurality of audio segments to an inference model performing inference to detect the non-semantic and paralinguistic event.
2. The method of claim 1 , further comprising:
detecting, by the inference model, the non-semantic and paralinguistic event; and
generating one or more metrics for the non-semantic and paralinguistic event.
3. The method of claim 2 , wherein the performing of the inference to detect the non-semantic and paralinguistic event occurs on a mobile device in substantial real-time.
4. The method of claim 1 , wherein the performing of the inference to detect the non-semantic and paralinguistic event comprises a determination of whether the audio stream contains speech from a person wearing a mask.
5. The method of claim 1 , wherein the non-semantic and paralinguistic event comprises one or more of crying, coughing, snoring, sneezing, wheezing, or a hiccup.
6. The method of claim 1 , wherein the non-semantic and paralinguistic event comprises coughing, and further comprising:
performing a cough-type classification to determine whether the cough comprises a wet cough, a dry cough, a cough associated with one or more of a respiratory tract infection, emphysema, exposure to smoke, exposure to air pollution, allergies, acid reflux, heart failure, or lung tumors.
7. The method of claim 1 , further comprising:
training the self-supervised triplet loss embedding model to learn the audio feature set in the self-supervised triplet loss manner from the plurality of speech audio clips from the speech dataset.
8. The method of claim 1 , further comprising:
training the student model based on a trained self-supervised triplet loss embedding model.
9. The method of claim 8 , wherein the student model is further configured based on one or more of: (i) varying a number filters in each layer of the student model, (ii) reducing a size of a bottleneck layer kernel by computing a global average over pixels in each output feature map, (iii) applying a compression operator to a bottleneck layer, wherein the compression operator is based on a Singular Value Decomposition (SVD) that is configured to learn a low-rank approximation of a weight matrix associated with the bottleneck layer, or (iv) applying Quantization-Aware training (QAT) that is configured to gradually reduce a numerical precision of weights associated with a bottleneck layer during training.
10. The method of claim 1 , wherein the generating of the embedding comprises utilizing temporal proximity in the plurality of speech audio clips as a self-supervision signal.
11. The method of claim 1 , further comprising:
instructing a user generating the audio stream to conduct a calibration procedure in which the user is instructed to utter the non-semantic and paralinguistic event N times;
computing an embedding for each utterance of non-semantic and paralinguistic event using the audio feature set;
computing a similarity between each pairwise combination of the N utterances; and
determining a verification threshold for the inference model based on the computed similarities.
12. The method of claim 11 , further comprising:
receiving user indication to detect one or more additional utterances of non-semantic and paralinguistic events over a time period;
initiating a recording of audio events during the time period;
terminating the recording of the audio events upon expiration of the time period; and
detecting the one or more additional utterances of non-semantic and paralinguistic events by comparing the recorded audio events with the N utterances.
13. The method of claim 12 , further comprising:
receiving a second user indication to provide the detected one or more additional utterances of non-semantic and paralinguistic events to a designated healthcare provider; and
responsive to the second user indication, providing the detected one or more additional utterances of non-semantic and paralinguistic events to the designated healthcare provider.
14. A computing device for detecting a non-semantic and paralinguistic event in an audio stream comprising:
one or more processors operable to perform operations, the operations comprising:
performing one or more pre-processing steps on the audio stream to generate an input audio sequence comprising a plurality of time-separated audio segments;
generating, by a student model, an embedding for the plurality of time-separated audio segments, the student model having been trained using knowledge distillation applied to a self-supervised triplet loss embedding model, the self-supervised triplet loss embedding model having been trained to learn an audio feature set in a self-supervised triplet loss manner from a plurality of speech audio clips; and
providing the embedding for the plurality of audio segments to an inference model performing inference to detect the non-semantic and paralinguistic event.
15. The computing device of claim 14 , wherein the computing device comprises one or more of a home intelligent assistant, a portable computer including a microphone for recording the audio stream, or a smartphone.
16. The computing device of claim 14 , wherein the performing of the inference to detect the non-semantic and paralinguistic event occurs on the computing device in substantial real-time.
17. The computing device of claim 14 , the operations further comprising:
training the self-supervised triplet loss embedding model to learn the audio feature set in the self-supervised triplet loss manner from the plurality of speech audio clips from the speech dataset.
18. The computing device of claim 14 , the operations further comprising:
training the student model based on a trained self-supervised triplet loss embedding model.
19. The computing device of claim 14 , the operations further comprising:
instructing a user generating the audio stream to conduct a calibration procedure in which the user is instructed to utter the non-semantic and paralinguistic event N times;
computing an embedding for each utterance of non-semantic and paralinguistic event using the audio feature set;
computing a similarity between each pairwise combination of the N utterances; and
determining a verification threshold for the inference model based on the computed similarities.
20. The computing device of claim 19 , the operations further comprising:
receiving user indication to detect one or more additional utterances of non-semantic and paralinguistic events over a time period;
initiating a recording of audio events during the time period;
terminating the recording of the audio events upon expiration of the time period; and
detecting the one or more additional utterances of non-semantic and paralinguistic events by comparing the recorded audio events with the N utterances.
21. The computing device of claim 20 , the operations further comprising:
receiving a second user indication to provide the detected one or more additional utterances of non-semantic and paralinguistic events to a designated healthcare provider; and
responsive to the second user indication, providing the detected one or more additional utterances of non-semantic and paralinguistic events to the designated healthcare provider.
22. One or more non-transitory computer-readable storage media for detecting a non-semantic and paralinguistic event in an audio stream comprising computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
performing one or more pre-processing steps on the audio stream to generate an input audio sequence comprising a plurality of time-separated audio segments;
generating, by a student model, an embedding for the plurality of time-separated audio segments, the student model having been trained using knowledge distillation applied to a self-supervised triplet loss embedding model, the self-supervised triplet loss embedding model having been trained to learn an audio feature set in a self-supervised triplet loss manner from a plurality of speech audio clips; and
providing the embedding for the plurality of audio segments to an inference model performing inference to detect the non-semantic and paralinguistic event.