Method, apparatus and system for monitoring health through audio data
According to some embodiments, an ear-worn device, e.g., a hearing aid, is provided that operates to monitor the health of a wearer of the ear-worn device based on an audio signal detected by the ear-worn device. In some embodiments, a method of monitoring the health of the wearer of the ear-worn device includes: detecting the audio signal with a microphone of the ear-worn device; processing the detected audio signal using a processor of the ear-worn device; and outputting, from an output signal generator of the ear-worn device, a generated output audio signal. In some embodiments, processing the detected audio signal includes: extracting data indicative of a health event associated with the wearer by processing the detected audio signal with a machine learning model; generating the output audio signal based on the detected audio signal; and outputting the data indicative of the health event.
1 . A method of monitoring health of a wearer of an ear-worn device based on an audio signal detected by the ear-worn device, the method comprising:
detecting the audio signal with a microphone of the ear-worn device;
processing the detected audio signal using a processor of the ear-worn device, the processing comprising:
extracting data indicative of a health event associated with the wearer of the ear-worn device by processing the detected audio signal with a machine learning model, wherein the machine learning model is configured to output a duration of the health event and/or a severity of the health event;
generating an output audio signal based on the detected audio signal; and
outputting the data indicative of the health event associated with the wearer of the ear-worn device; and
outputting, from an output signal generator of the ear-worn device, the generated output audio signal.
2 . The method of claim 1 , wherein the ear-worn device is a hearing aid.
3 . The method of claim 1 , wherein the health event comprises coughing, sneezing, wheezing, snoring, chewing, swallowing, drinking, and/or speaking.
4 . The method of claim 1 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal using a neural network.
5 . The method of claim 4 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal with a recurrent neural network or a convolutional neural network.
6 . The method of claim 1 , wherein processing the detected audio signal with the machine learning model comprises:
detecting the health event by processing the detected audio signal with a detector machine learning model; and
after detecting the health event with the detector machine learning model, characterizing the health event by processing the detected audio signal with a characterization machine learning model, wherein the detector machine learning model is different from the characterization machine learning model.
7 . The method of claim 6 , wherein the characterization machine learning model is configured to output the duration of the health event and/or the severity of the health event.
8 . The method of claim 1 , wherein the extracted data indicative of the health event comprises a portion of the detected audio signal corresponding to the health event, an indication that the health event occurred, and/or a characterization of the health event.
9 . The method of claim 1 , wherein outputting the data indicative of the health event comprises storing the data indicative of the health event in a memory of the ear-worn device.
10 . The method of claim 1 , wherein outputting the data indicative of the health event comprises transmitting, to a computing device different from the ear-worn device, the data indicative of the health event.
11 . The method of claim 10 , wherein outputting the data indicative of the health event comprises transmitting the data indicative of the health event using a wireless communication protocol.
12 . The method of claim 1 , wherein generating the output audio signal based on the detected audio signal comprises removing background noise from the detected audio signal.
13 . The method of claim 12 , wherein removing the background noise from the detected audio signal comprises generating a processed audio signal, and
wherein processing the detected audio signal with the machine learning model comprises processing the processed audio signal with the machine learning model.
14 . The method of claim 12 , wherein generating the output audio signal based on the detected audio signal further comprises, after removing the background noise from the detected audio signal, amplifying at least one component of the detected audio signal.
15 . The method of claim 1 ,
wherein generating the output audio signal based on the detected audio signal comprises estimating a signal-to-noise ratio (SNR) of the detected audio signal, and
wherein extracting the data indicative of the health event associated with the wearer of the ear-worn device comprises processing the detected audio signal with the machine learning model only when the estimated SNR is within a specified range.
16 . The method of claim 1 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal with a first machine learning model, and wherein generating the output audio signal based on the detected audio signal comprises:
isolating a component of the detected audio signal representing a voice of the wearer from among temporally overlapping voice components from multiple speakers by processing the detected audio signal with a second machine learning model using a voice signature of the wearer.
17 . The method of claim 16 , wherein processing the detected audio signal with the first machine learning model comprises processing only the isolated component of the detected audio signal with the first machine learning model.
18 . The method of claim 1 , wherein the machine learning model comprises a first machine learning model, and wherein generating the output audio signal comprises processing the detected audio signal with a second machine learning model.
19 . The method of claim 18 , wherein processing the detected audio signal with the second machine learning model comprises processing the detected audio signal with a neural network.
20 . The method of claim 1 , wherein processing the detected audio signal using the processor of the ear-worn device further comprises computing a Fast Fourier Transform (FFT) of the detected audio signal.
21 . The method of claim 20 , wherein computing the FFT of the detected audio signal comprises computing the FFT prior to:
extracting the data indicative of the health event associated with the wearer of the ear-worn device by processing the detected audio signal with the machine learning model; and
generating the output audio signal based on the detected audio signal.
22 . An ear-worn device configured to monitor health of a wearer of the ear-worn device based on an audio signal detected by the ear-worn device, the ear-worn device comprising:
a microphone configured to detect the audio signal;
a processor;
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the processor, cause the processor to perform a method comprising:
extracting data indicative of a health event associated with the wearer of the ear-worn device by processing the detected audio signal with a machine learning model, wherein the machine learning model is configured to output a duration of the health event and/or a severity of the health event;
generating an output audio signal based on the detected audio signal; and
outputting the data indicative of the health event associated with the wearer of the ear-worn device; and
an output signal generator configured to output the generated output audio signal.
23 . The ear-worn device of claim 22 , wherein the ear-worn device is a hearing aid.
24 . The ear-worn device of claim 22 , wherein the health event comprises coughing, sneezing, wheezing, snoring, chewing, swallowing, and/or drinking.
25 . The ear-worn device of claim 22 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal using a neural network.
26 . The ear-worn device of claim 25 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal with a recurrent neural network or a convolutional neural network.
27 . The ear-worn device of claim 22 , wherein processing the detected audio signal with the machine learning model comprises:
detecting the health event by processing the detected audio signal with a detector machine learning model; and
after detecting the health event with the detector machine learning model, characterizing the health event by processing the detected audio signal with a characterization machine learning model, wherein the detector machine learning model is different from the characterization machine learning model.
28 . The ear-worn device of claim 27 , wherein the characterization machine learning model is configured to output the duration of the health event and/or the severity of the health event.
29 . The ear-worn device of claim 22 , wherein the extracted data indicative of the health event comprises a portion of the detected audio signal corresponding to the health event, an indication that the health event occurred, and/or a characterization of the health event.
30 . The ear-worn device of claim 22 , wherein outputting the data indicative of the health event comprises storing the data indicative of the health event in a memory of the ear-worn device.
31 . The ear-worn device of claim 22 , wherein outputting the data indicative of the health event comprises transmitting, to a computing device different from the ear-worn device, the data indicative of the health event.
32 . The ear-worn device of claim 31 , wherein outputting the data indicative of the health event comprises transmitting the data indicative of the health event using a wireless communication protocol.
33 . The ear-worn device of claim 22 , wherein generating the output audio signal based on the detected audio signal comprises removing background noise from the detected audio signal.
34 . The ear-worn device of claim 33 , wherein removing the background noise from the detected audio signal comprises generating a processed audio signal, and
wherein processing the detected audio signal with the machine learning model comprises processing the processed audio signal with the machine learning model.
35 . The ear-worn device of claim 33 , wherein generating the output audio signal based on the detected audio signal further comprises, after removing the background noise from the detected audio signal, amplifying at least one component of the detected audio signal.
36 . The ear-worn device of claim 22 ,
wherein generating the output audio signal based on the detected audio signal comprises estimating a signal-to-noise ratio (SNR) of the detected audio signal, and
wherein extracting the data indicative of the health event associated with the wearer of the ear-worn device comprises processing the detected audio signal with the machine learning model only when the estimated SNR is within a specified range.
37 . The ear-worn device of claim 22 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal with a first machine learning model, and wherein generating the output audio signal based on the detected audio signal comprises:
isolating a component of the detected audio signal representing a voice of the wearer from among temporally overlapping voice components from multiple speakers by processing the detected audio signal with a second machine learning model using a voice signature of the wearer.
38 . The ear-worn device of claim 37 , wherein processing the detected audio signal with the first machine learning model to extract data indicative of the health event associated with the wearer of the ear-worn device comprises processing only the isolated component of the detected audio signal with the first machine learning model.
39 . The ear-worn device of claim 22 , wherein the machine learning model comprises a first machine learning model, and wherein generating the output audio signal comprises processing the detected audio signal with a second machine learning model.
40 . The ear-worn device of claim 39 , wherein processing the detected audio signal with the second machine learning model comprises processing the detected audio signal with a neural network.
41 . The ear-worn device of claim 22 , wherein the method further comprises computing a Fast Fourier Transform (FFT) of the detected audio signal.
42 . The ear-worn device of claim 41 , wherein computing the FFT of the detected audio signal comprises computing the FFT prior to:
extracting the data indicative of the health event associated with the wearer of the ear-worn device by processing the detected audio signal with the machine learning model; and
generating the output audio signal based on the detected audio signal.
43 . At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by a processor, cause the processor to perform a method of monitoring health of a wearer of an ear-worn device based on an audio signal detected by the ear-worn device, the method comprising:
extracting data indicative of a health event associated with the wearer of the ear-worn device by processing the detected audio signal with a machine learning model, wherein the machine learning model is configured to output a duration of the health event and/or a severity of the health event;
generating an output audio signal based on the detected audio signal;
outputting the data indicative of the health event associated with the wearer of the ear-worn device; and
transmitting the output audio signal to an output signal generator of the ear-worn device.
44 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein the ear-worn device is a hearing aid.
45 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein the health event comprises coughing, sneezing, wheezing, snoring, chewing, swallowing, and/or drinking.
46 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal using a neural network.
47 . The at least one non-transitory computer-readable storage medium of claim 46 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal with a recurrent neural network or a convolutional neural network.
48 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein processing the detected audio signal with the machine learning model comprises:
detecting the health event by processing the detected audio signal with a detector machine learning model; and
after detecting the health event with the detector machine learning model, characterizing the health event by processing the detected audio signal with a characterization machine learning model, wherein the detector machine learning model is different from the characterization machine learning model.
49 . The at least one non-transitory computer-readable storage medium of claim 48 , wherein the characterization machine learning model is configured to output the duration of the health event and/or the severity of the health event.
50 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein the extracted data indicative of the health event comprises a portion of the detected audio signal corresponding to the health event, an indication that the health event occurred, and/or a characterization of the health event.
51 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein outputting the data indicative of the health event comprises storing the data indicative of the health event in a memory of the ear-worn device.
52 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein outputting the data indicative of the health event comprises transmitting, to a computing device different from the ear-worn device, the data indicative of the health event.
53 . The at least one non-transitory computer-readable storage medium of claim 52 , wherein outputting the data indicative of the health event comprises transmitting the data indicative of the health event using a wireless communication protocol.
54 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein generating the output audio signal based on the detected audio signal comprises removing background noise from the detected audio signal.
55 . The at least one non-transitory computer-readable storage medium of claim 54 , wherein removing the background noise from the detected audio signal comprises generating a processed audio signal, and
wherein processing the detected audio signal with the machine learning model comprises processing the processed audio signal with the machine learning model.
56 . The at least one non-transitory computer-readable storage medium of claim 54 , wherein generating the output audio signal based on the detected audio signal further comprises, after removing the background noise from the detected audio signal, amplifying at least one component of the detected audio signal.
57 . The at least one non-transitory computer-readable storage medium of claim 43 ,
wherein generating the output audio signal based on the detected audio signal comprises estimating a signal-to-noise ratio (SNR) of the detected audio signal, and
wherein extracting the data indicative of the health event associated with the wearer of the ear-worn device comprises processing the detected audio signal with the machine learning model only when the estimated SNR is within a specified range.
58 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein processing the detected audio signal with the machine learning model comprises processing the detected audio signal with a first machine learning model, and wherein generating the output audio signal based on the detected audio signal comprises:
isolating a component of the detected audio signal representing a voice of the wearer from among temporally overlapping voice components from multiple speakers by processing the detected audio signal with a second machine learning model using a voice signature of the wearer.
59 . The at least one non-transitory computer-readable storage medium of claim 58 , wherein processing the detected audio signal with the first machine learning model to extract data indicative of the health event associated with the wearer of the ear-worn device comprises processing only the isolated component of the detected audio signal with the first machine learning model.
60 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein the machine learning model comprises a first machine learning model, and wherein generating the output audio signal comprises processing the detected audio signal with a second machine learning model.
61 . The at least one non-transitory computer-readable storage medium of claim 60 , wherein processing the detected audio signal with the second machine learning model comprises processing the detected audio signal with a neural network.
62 . The at least one non-transitory computer-readable storage medium of claim 43 , the method further comprises computing a Fast Fourier Transform (FFT) of the detected audio signal.
63 . The at least one non-transitory computer-readable storage medium of claim 62 , wherein computing the FFT of the detected audio signal comprises computing the FFT prior to:
extracting the data indicative of the health event associated with the wearer of the ear-worn device by processing the detected audio signal with the machine learning model; and
generating the output audio signal based on the detected audio signal.
64 . The method of claim 1 , wherein the machine learning model is further configured to output a type of the health event.
65 . The ear-worn device of claim 22 , wherein the machine learning model is further configured to output a type of the health event.
66 . The at least one non-transitory computer-readable storage medium of claim 43 , wherein the machine learning model is further configured to output a type of the health event.