Identifying the position of a horn honk or other acoustical information using multiple autonomous vehicles
The technology relates to determining a source of a horn honk or other noise in an environment around one or more self-driving vehicles. Aspects of the technology leverage real-time information from a group of self-driving vehicles regarding received acoustical information. The location and pose of each self-driving vehicle in the group, along with the precise arrangement of acoustical sensors on each vehicle, can be used to triangulate or otherwise identify the actual location in the environment for the origin of the horn honk or other sound. Other sensor information, map data, and additional data can be used narrow down or refine the location of a likely noise source. Once the location and source of the noise is known, each self-driving vehicle can use that information to modify current driving operations and/or use it as part of a reinforcement learning approach for future driving situations.
1. A method comprising:
obtaining, by a perception system of a vehicle, audio sensor data of a sound emanated in an external environment around the vehicle;
responsive to a change in the sound, receiving, by one or more processors of the vehicle, other audio sensor data of the sound from one or more other vehicles, the other audio sensor data including a direction-of-arrival and a timestamp;
determining, by the one or more processors based on (a) the audio sensor data, (b) the other audio sensor data, (c) a location of the vehicle at the timestamp, and (d) respective locations of the one or more other vehicles at the timestamp, a particular object from which the sound emanated; and
modifying, by the one or more processors based on the particular object, a driving operation of the vehicle operating in an autonomous driving mode.
2. The method of claim 1 , wherein the particular object is an emergency vehicle.
3. The method of claim 2 , wherein the emergency vehicle is one of: an ambulance, a fire truck or police car.
4. The method of claim 2 , wherein modifying the driving operation includes at least one of: slowing down for the emergency vehicle, changing lanes for the emergency vehicle, or pulling over for the emergency vehicle.
5. The method of claim 1 , further comprising:
obtaining, by the perception system, imagery of the external environment around the vehicle; and
determining, by the one or more processors based on the imagery and the audio sensor data, whether the particular object is an emergency vehicle.
6. The method of claim 5 , wherein determining whether the particular object is the emergency vehicle includes determining, by the one or more processors based on the imagery, whether lights associated with the particular object are lights associated with the emergency vehicle.
7. The method of claim 1 , wherein the particular object is a person.
8. The method of claim 7 , wherein the person is associated with an emergency vehicle.
9. The method of claim 8 , wherein the sound includes audio information from the person.
10. The method of claim 8 , wherein the emergency vehicle is of: an ambulance, a fire truck or police car.
11. The method of claim 1 , wherein the sound includes at least two of: a siren associated with an emergency vehicle, a horn honk, or audio information from a person.
12. The method of claim 11 , further comprising:
obtaining, by the perception system, imagery of the external environment around the vehicle; and
determining, by the one or more processors based on the imagery, whether lights associated with the particular object are lights associated with the emergency vehicle.
13. A method comprising:
receiving, by a microphone array of a vehicle operating in an autonomous driving mode, audio sensor data of a sound emanated in an external environment around the vehicle, the audio sensor data including a direction-of-arrival and a timestamp;
determining, by one or more processors of the vehicle based on the audio sensor data, a particular object from which the sound emanated;
determining, by the one or more processors based on (a) the audio sensor data, (b) a location of the vehicle at the timestamp, and (c) respective locations of one or more vehicles at the timestamp, whether the sound is directed at the vehicle operating in the autonomous driving mode or another road agent; and
modifying, by the one or more processors based on the particular object and the determining whether the sound is directed at the vehicle or the other road agent, a driving operation of the vehicle.
14. The method of claim 13 , wherein the vehicle and the other road agent are associated with a fleet of vehicles.
15. The method of claim 13 , further comprising receiving, by the one or more processors from a back-end system via a network, other audio sensor data of the sound from the other road agent, the other audio sensor data including another direction-of-arrival; and
wherein determining the particular object from which the sound emanated is further based on the other audio sensor data and a location of the other road agent.
16. The method of claim 13 , further comprising determining whether the sound is indicative of a problematic or dangerous condition on a roadway.
17. The method of claim 16 , further comprising determining whether the particular object is an emergency vehicle.
18. The method of claim 16 , further comprising determining whether the particular object is a person associated with an emergency vehicle.
19. The method of claim 13 , wherein:
the sound is a horn honk, and
the method further comprises:
determining, by the one or more processors based on at least one of: a timing, a frequency, a harmony, a pitch change, or an amplitude of the horn honk, whether the horn honk is a first type of horn honk or a second type of horn honk;
responsive to determining that the horn honk is the first type of horn honk, selecting, by the one or more processors, a first behavior model for predicting a behavior of another vehicle; and
responsive to determining that the sound is the second type of horn honk, selecting, by the one or more processors, a second behavior model for predicting the behavior of the other vehicle, the second behavior model being different from the first behavior model.
20. The method of claim 19 , wherein modifying the driving operation is further based on whether the first behavior model or the second behavior model is selected.
21. The method of claim 20 , wherein modifying the driving operation includes, responsive to selecting the second behavior model, at least one of: slowing down for the other vehicle, changing lanes for the other vehicle, or pulling over for the other vehicle.