Emergency sound localization
Techniques for determining information associated with sounds detected in an environment based on audio data are discussed herein. Audio sensors of a vehicle may determine audio data associated with sounds from the environment. Sounds may be caused by objects in the environment such as emergency vehicles, construction zones, non-emergency vehicles, humans, audio speakers, nature, etc. A model may determine a classification of the audio data and/or a probability value representing a likelihood that sound in the audio data is associated with the classification. A direction of arrival may be determined based on receiving classification values from multiple audio sensors of the vehicle, and other actions can be performed or the vehicle can be controlled based on the direction of arrival.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving first audio data from a first audio sensor associated with a vehicle;
receiving second audio data from a second audio sensor associated with the vehicle;
inputting the first audio data and the second audio data into a model;
receiving, from the model, a first classification of the first audio data, the first classification including a first probability that the first audio data is associated with a sound class;
receiving, from the model, a second classification of the second audio data, the second classification including a second probability that the second audio data is associated with the sound class; and
based at least in part on the first probability that the first audio data is associated with the sound class and the second probability that the second audio data is associated with the sound class, at least one of:
determining a direction of arrival associated with a source of the first audio data or the second audio data; or
determining a level of computational processing associated with another model for determining data for the source of the first audio data or the second audio data,
wherein the first classification and second classification comprises an emergency event or a non-emergency event.
2. The system of claim 1 , wherein the first classification comprises a first classification value representing a likelihood that the first audio data is associated with the first classification and the second classification comprises a second classification value representing a likelihood that the second audio data is associated with the second classification, and
wherein the operations further comprise determining, based at least in part on a difference between the first classification and the second classification, the direction of arrival associated with the first audio data or the second audio data.
3. The system of claim 2 , the operations further comprising:
receiving, as pose data, position data and orientation data associated the first audio sensor; and
determining, based at least in part on the first classification value, the second classification value, and the pose data, the direction of arrival associated with the source of the first audio data or the second audio data.
4. The system of claim 3 , the operations further comprising:
determining, based at least in part on the first classification of the second classification, that the first audio data or the second audio data is associated with the emergency event; and
determining an action to control the vehicle in an environment based at least in part on the direction of arrival associated with the emergency event.
5. The system of claim 1 , wherein the sound class comprises one or more of: an emergency sound class, a non-emergency sound class, an engine sound class, a music sound class, a thundering sound class, an animal sound class, an ambulance siren sound class, a police siren sound class, a fire truck siren sound class, a power tool sound class, a reverse direction sound class, a honking sound class or a speech sound class.
6. A method comprising:
receiving first audio data;
receiving second audio data;
inputting the first audio data and the second audio data into a model comprising one or more classifiers;
receiving, from the model, a first classification value indicative of a likelihood that the first audio data is associated with a classifier of the one or more classifiers;
receiving, from the model, a second classification value indicative of a likelihood that the second audio data is associated with the classifier of the one or more classifiers; and
based at least in part on the first classification value and the second classification value, at least one of:
determining a direction of arrival value associated with a source of the first audio data or the second audio data; or
determining a level of computational processing associated with another model for determining data for the source of the first audio data or the second audio data,
wherein the first classification and second classification comprises an emergency event or a non-emergency event.
7. The method of claim 6 , wherein the classifier comprises one or more of: an emergency sound class, a non-emergency sound class, an engine sound class, a music sound class, a thundering sound class, an animal sound class, an ambulance siren sound class, a police siren sound class, a fire truck siren sound class, a power tool sound class, a reverse direction sound class, a honking sound class or a speech sound class.
8. The method of claim 6 , further comprising:
based at least in part on determining the level of computational processing, sending the first classification value and the second classification value to the other model; and
causing the other model to determine a threshold associated with a sensor based at least in part on the first classification value and the second classification value.
9. The method of claim 6 , wherein determining the level of computational processing comprises determining an amount of computational processing or memory capacity to allocate during subsequent processing of an object associated with the first audio data or the second audio data.
10. The method of claim 6 , further comprising:
determining an action to control a vehicle in an environment based at least in part on the direction of arrival value associated with the source of the first audio data or the second audio data.
11. The method of claim 9 , wherein:
the first audio data is associated with a first audio sensor,
the second audio data is associated with a second audio sensor, and
determining the direction of arrival value associated with the first audio data or the second audio data is further based at least in part on one or more of position data or orientation data associated with at least one of the first audio sensor or the second audio sensor.
12. The method of claim 9 , wherein the direction of arrival value is a first direction of arrival value, the method further comprising:
determining, based at least in part on at least one of: a magnitude difference between the first audio data and the second audio data or a time difference between the first audio data and the second audio data, a second direction of arrival value.
13. The method of claim 6 , wherein:
the first audio data is associated with a first signal captured by a first audio sensor and a second signal captured by a second audio sensor, and
the second audio data is associated with a third signal captured by a third audio sensor and a fourth signal captured by a fourth audio sensor.
14. The method of claim 10 , further comprising:
determining that the first audio data or the second audio data are associated with the emergency event; and
wherein controlling the vehicle comprises at least one of stopping the vehicle or determining a trajectory for the vehicle to follow in the environment.
15. The method of claim 6 , further comprising:
identifying an intensity or a frequency of sound associated with the first audio data and the second audio data;
determining the first classification value based at least in part on the intensity or the frequency of sound associated with the first audio data; and
determining the second classification value based at least in part on the intensity or the frequency of sound associated with the second audio data.
16. The method of claim 6 , wherein the model is a first model and the direction of arrival value is a first direction of arrival value, further comprising:
sending information associated with the first classification value and the second classification value to a second model; and
receiving, from the second model, one or more of:
a second direction of arrival value associated with the source of the first audio data or the second audio data;
a probability that the first audio data and the second audio data are associated with a same class;
a probability that the first audio data and the second audio data are associated with a same object in an environment; or
a probability that the first audio data is associated with a first object in an environment and the second audio data is associated with a second object in the environment.
17. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving first audio data;
receiving second audio data;
inputting the first audio data and the second audio data into a model comprising one or more classifiers;
receiving, from the model, a first classification value indicative of a likelihood that the first audio data is associated with a classifier of the one or more classifiers;
receiving, from the model, a second classification value indicative of a likelihood that the second audio data is associated with the classifier of the one or more classifiers; and
based at least in part on the first classification value and the second classification value, at least one of:
determining a direction of arrival value associated with a source of the first audio data or the second audio data; or
determining a level of computational processing associated with another model for determining data for the source of the first audio data or the second audio data,
wherein the first classification and second classification comprises an emergency event or a non-emergency event.
18. The one or more non-transitory computer-readable media of claim 17 , the operations further comprising:
determining that the first audio data or the second audio data is associated with the emergency event; and
determining the direction of arrival value associated with the emergency event.
19. The one or more non-transitory computer-readable media of claim 17 , the operations further comprising:
receiving, as pose data, one or more of position data or orientation data associated a first audio sensor associated with the first audio data; and
determining, based at least in part on the first classification value, the second classification value, and the pose data, the direction of arrival value associated with the first audio data or the second audio data.
20. The one or more non-transitory computer-readable media of claim 17 , the operations further comprising:
based at least in part on determining the level of computational processing, sending the first classification value and the second classification value to the other model; and
causing the other model to determine a threshold associated with a sensor based at least in part on the first classification value and the second classification value.