Sound detection for electronic devices
The subject disclosure provides systems and methods for providing locally trained models for detecting individual sounds using electronic devices. Local detection of individual sounds with a detection model at an electronic device can be provided by obtaining training samples for the detection model with the electronic device, and generating additional negative and positive training samples based on the obtained training samples. A two-stage detection process may be provided, in which a trigger model at a device compares an audio input to a reference sound to trigger a detection model at the device. The detection of individual sounds with a detection model at an electronic device can also leverage audio capture capabilities of multiple devices in an acoustic scene to capture multiple concurrent training samples.
1. A method, comprising:
obtaining, with an electronic device, a sound input;
providing an audio signal based on the sound input to a first classifier at the electronic device;
determining, with the first classifier, that the audio signal meets a similarity threshold based on a comparison of the audio signal with a device-generated class prototype;
providing, based on the determining with the first classifier, the audio signal to a second classifier at the electronic device, the second classifier having been trained to detect a specific sound using at least one training sound sample captured by the electronic device and at least one device-generated training sound sample generated by the electronic device based on the at least one training sound sample; and
generating, by the second classifier, an output indicating whether a specific sound is detected in the sound input.
2. The method of claim 1 , further comprising generating an alert with the electronic device if the output of the second classifier indicates that the specific sound is detected.
3. The method of claim 1 , wherein the at least one device-generated training sound sample includes:
at least one negative training sample generated by the electronic device based on a first modification of the at least one training sound sample, and
at least one positive training sample generated by the electronic device based on a second modification the at least one training sound sample.
4. The method of claim 3 , further comprising training the second classifier at the electronic device by providing the at least one training sound sample, the at least one negative training sample, and the at least one positive training sample as inputs to the second classifier.
5. The method of claim 4 , wherein the at least one training sound sample comprises a plurality of training sound samples, the method further comprising generating the device-generated class prototype at the electronic device, in part, by:
segmenting of each of the plurality of training sound samples to generate a corresponding plurality of training segments;
selecting a subset of the plurality of training segments based on a similarity of the training segments of the subset; and
aligning the subset of the plurality of training segments to generate a set of aligned training segments.
6. The method of claim 5 , further comprising obtaining at least a first one of the plurality of training sound samples using a microphone of the electronic device.
7. The method of claim 6 , further comprising obtaining at least a second one of the plurality of training sound samples from another electronic device.
8. The method of claim 1 , wherein the comparison of the audio signal with the device-generated class prototype comprises a determination of a cosine distance between a representation of the audio signal and the device-generated class prototype.
9. An electronic device, comprising:
a memory storing:
a trigger model,
a reference sound generated at the electronic device, and
a detection model trained to detect a specific sound using at least one training sound sample captured by the electronic device and at least one device-generated training sound sample generated by the electronic device based on the at least one training sound sample; and
one or more processors, wherein the one or more processors are configured to:
obtain a sound input;
trigger the detection model by comparing, with the trigger model, an audio signal based on the sound input with the reference sound; and
determine whether the sound input includes the specific sound by providing the audio signal to the detection model.
10. The electronic device of claim 9 , wherein the one or more processors are further configured to generate the reference sound based on a plurality of training inputs to the electronic device.
11. The electronic device of claim 9 , further comprising an output component configured to generate a notification responsive to a determination, by the detection model, that the sound input includes the specific sound.
12. The electronic device of claim 9 , further comprising a microphone configured to obtain the sound input.
13. The electronic device of claim 12 , wherein the one or more processors are further configured to:
obtain the at least one training sound sample using the microphone; and
generate the reference sound based on the at least one training sound sample.
14. The electronic device of claim 13 , wherein the one or more processors are further configured to:
generate the at least one device-generated training sound sample by:
generating at least one negative training sample by applying a first modification of the at least one training sound sample, and
generate at least one positive training sample by applying a second modification to the at least one training sound sample.
15. The electronic device of claim 14 , wherein the one or more processors are further configured to train the detection model by providing the at least one training sound sample, the at least one negative training sample, and the at least one positive training sample to the detection model as training inputs.
16. The electronic device of claim 13 , wherein the one or more processors are further configured to obtain at least one additional training sound sample from another electronic device.
17. An electronic device, comprising:
a memory; and
one or more processors configured to:
obtain a sound input;
provide an audio signal based on the sound input to a first classifier at the electronic device;
determine, with the first classifier, that the audio signal meets a similarity threshold based on a comparison of the audio signal with a device-generated class prototype;
provide, based on the determining with the first classifier, the audio signal to a second classifier at the electronic device, the second classifier having been trained to detect a specific sound using at least one training sound sample captured by the electronic device and at least one device-generated training sound sample generated by the electronic device based on the at least one training sound sample; and
generate, by the second classifier, an output indicating whether the specific sound is detected in the sound input.
18. The electronic device of claim 17 , wherein the one or more processors are further configured to generate an alert with the electronic device if the output of the second classifier indicates that the specific sound is detected.
19. The electronic device of claim 17 , wherein the at least one device-generated training sound sample includes:
at least one negative training sample generated by the electronic device based on a first modification of the at least one training sound sample, and
at least one positive training sample generated by the electronic device based on a second modification the at least one training sound sample.
20. The electronic device of claim 19 , wherein the one or more processors are further configured to train the second classifier at the electronic device by providing the at least one training sound sample, the at least one negative training sample, and the at least one positive training sample as inputs to the second classifier.
21. The electronic device of claim 20 , wherein the at least one training sound sample comprises a plurality of training sound samples, and wherein the one or more processors are further configured to generate the device-generated class prototype at the electronic device, in part, by:
segmenting of each of the plurality of training sound samples to generate a corresponding plurality of training segments;
selecting a subset of the plurality of training segments based on a similarity of the training segments of the subset; and
aligning the subset of the plurality of training segments to generate a set of aligned training segments.
22. The electronic device of claim 21 , wherein the one or more processors are further configured to obtain at least a first one of the plurality of training sound samples using a microphone of the electronic device.
23. The electronic device of claim 22 , wherein the one or more processors are further configured to obtain at least a second one of the plurality of training sound samples from another electronic device.
24. The electronic device of claim 23 , wherein the electronic device is associated with an account and the other electronic device is also associated with the account.
25. The electronic device of claim 17 , wherein the comparison of the audio signal with the device-generated class prototype comprises a determination of a cosine distance between a representation of the audio signal and the device-generated class prototype.
26. The electronic device of claim 17 , wherein the first classifier comprises a trigger model and the second classifier comprises a detection model.
27. The electronic device of claim 26 , wherein the one or more processors are further configured to provide, to the detection model along with the audio signal, a trigger signal to activate the detection model to generate the output indicating whether the specific sound is detected in the sound input.
28. The electronic device of claim 17 , wherein the specific sound comprises a sound that has been previously enrolled for detection by the electronic device.