Audio detection and monitoring of respiration
The subject technology provides a framework for estimating respiratory rates from audio data recordings. A multi-task learning network may be trained to output respiratory rates, breathing conditions, and/or noise conditions based on input audio data recordings. The audio data recordings may be generated using wearable audio devices with near-field microphones. The respiratory rates may be provided along with other workout information by a health application of an electronic device. Additional sensor data and/or health data may be used in combination with the audio data and/or the respiratory rates and/or breathing conditions for respiratory and/or other health monitoring by an electronic device.
1 . A method comprising:
obtaining at least one audio recording of at least one breath cycle of a user;
providing audio input data corresponding to the at least one audio recording to a multi-task recurrent network trained using a multi-task objective function to generate a respiratory rate and an indication of a breathing condition from a breath embedding layer of the multi-task recurrent network and an indication of a background noise of the at least one audio recording from a noise embedding layer of the multi-task recurrent network; and
obtaining, as an output from the multi-task recurrent network, the indication of the background noise of the at least one audio recording from the noise embedding layer and at least one of the respiratory rate of the user or the indication of the breathing condition of the user.
2 . The method of claim 1 , wherein obtaining the at least one audio recording of the at least one breath cycle of the user comprises obtaining multiple audio recordings each having a duration of between three seconds and ten seconds.
3 . The method of claim 2 , wherein obtaining the multiple audio recordings comprises obtaining the multiple audio recordings with a near-field microphone of a wearable audio device.
4 . The method of claim 3 , wherein obtaining the multiple audio recordings comprises obtaining the multiple audio recordings responsive to receiving a trigger from a health application on an electronic device.
5 . The method of claim 4 , further comprising, with a speaker of the wearable audio device, outputting audio content from the electronic device.
6 . The method of claim 5 , wherein obtaining the multiple audio recordings comprises obtaining the multiple audio recordings with the wearable audio device while outputting the audio content with the wearable audio device.
7 . The method of claim 4 , wherein providing the audio input data to the multi-task recurrent network comprises providing the audio input data to the multi-task recurrent network at the wearable audio device.
8 . The method of claim 4 , further comprising providing the multiple audio recordings to the electronic device, wherein providing the audio input data to the multi-task recurrent network comprises providing the audio input data to the multi-task recurrent network at the electronic device.
9 . The method of claim 4 , further comprising providing the multiple audio recordings to an additional electronic device, wherein providing the audio input data to the multi-task recurrent network comprises providing the audio input data to the multi-task recurrent network at the additional electronic device.
10 . The method of claim 9 , wherein the electronic device comprises a smart watch and wherein the additional electronic device comprises a smartphone.
11 . The method of claim 4 , further comprising obtaining additional sensor data with a sensor of the electronic device.
12 . The method of claim 11 , wherein the additional sensor data comprises multiple heart rate measurements each corresponding to one of the multiple audio recordings.
13 . The method of claim 4 , wherein obtaining the multiple audio recordings comprises obtaining the multiple audio recordings at a corresponding plurality of times associated with a recording of a workout by the electronic device.
14 . The method of claim 13 , wherein the corresponding plurality of times include at least one time within a minute prior to an end time of the workout and at least one time within a minute after the end time of the workout and the method further comprising:
providing the respiratory rate for display in association with additional information for the workout by a user interface of the health application.
15 . The method of claim 1 , wherein the multi-task objective function comprises a loss function.
16 . The method of claim 1 , wherein the indication of the breathing condition of the user comprises a classification indicating a normal breathing condition or a heavy breathing condition of the user.
17 . The method of claim 1 , wherein the indication of the background noise of the at least one audio recording comprises a classification indicating a noise condition or a noiseless condition of an environment of the user.
18 . The method of claim 17 , wherein obtaining the respiratory rate and the breathing condition comprise obtaining the respiratory rate and the breathing condition with the breath embedding layer of the multi-task recurrent network, and wherein obtaining the noise condition of the at least one audio recording comprises obtaining the noise condition with the noise embedding layer of the multi-task recurrent network.
19 . The method of claim 1 , further comprising generating the audio input data by generating at least one spectrogram from at least one audio recording.
20 . The method of claim 19 , wherein generating the audio input data further comprises performing a time-convolution of the at least one spectrogram.
21 . The method of claim 19 , further comprising, prior to generating the at least one spectrogram:
determining whether a portion of the at least one audio recording includes a recording of human speech; and
discarding any portion of the at least one recording that includes the recording of the human speech.
22 . An electronic device, comprising:
a memory storing a multi-task recurrent network that includes a breath embedding layer and a noise embedding layer; and
one or more processors configured to:
obtain at least one audio recording of at least one breath cycle of a user from an audio device that is configured to output audio content provided by the electronic device;
provide audio input data corresponding to the at least one audio recording to the multi-task recurrent network trained using a multi-task objective function to generate a respiratory rate, an indication of a breathing condition from a breath embedding layer of the multi-task recurrent network and an indication of a background noise of the at least one audio recording from a noise embedding layer of the multi-task recurrent network;
obtain, as an output from the multi-task recurrent network, an indication of a background noise of the at least one audio recording from the noise embedding layer and at least one of the respiratory rate of the user or the indication of a breathing condition of the user; and
provide the indication of the background noise and at least one of the respiratory rate or the breathing condition for display by a display of the electronic device.
23 . The electronic device of claim 22 , wherein the audio device comprises a pair of wearable earbuds that are wirelessly paired to the electronic device.
24 . The electronic device of claim 22 , wherein the one or more processors are configured to provide the respiratory rate for display in connection with a workout using a health application at the electronic device.
25 . The electronic device of claim 24 , wherein the one or more processors are further configured to receive additional health data from another electronic device in connection with the workout.
26 . The electronic device of claim 25 , wherein the another electronic device comprises a wearable electronic device comprising at least one sensor, and wherein the additional health data comprises a pulse rate or a blood oxygen level determined using the at least one sensor.
27 . A method of training a multi-task recurrent network for estimation of respiratory rates from audio data, the method comprising:
providing input training data to the multi-task recurrent network, the input training data corresponding to a plurality of audio recordings;
generating multiple training outputs with the multi-task recurrent network responsive to the input training data; and
adjusting a plurality of parameters of a breath embedding layer of the multi-task recurrent network and a plurality of parameters of a noise embedding layer of the multi-task recurrent network using a multi-task objective function to compare the multiple training outputs of the multi-task recurrent network to output training data, the output training data including a known respiratory rate, a known breathing condition, and a known noise condition associated with the plurality of audio recordings.
28 . The method of claim 27 , wherein the multi-task objective function comprises a concordance correlation coefficient loss, a breath cross-entropy loss, and a noise cross-entropy loss.