Autonomous vehicle driving path selection in the presence of emergency vehicle sounds

1. A method comprising:

obtaining, using one or more audio detectors of a vehicle, a sound recording comprising a sound emitted by an emergency vehicle (EV);

applying, using a processing device, a sound localization (SL) model to the sound recording to generate an SL output, wherein the SL output comprises a first map of possible locations of the EV in a driving environment of the vehicle;

simulating, by the processing device and using the SL output, trajectories of one or more simulated EVs in the driving environment of the vehicle;

causing, responsive to proximity of one or more of the simulated trajectories to a driving path of the vehicle, the driving path of the vehicle to be modified; and

controlling the vehicle based on the modified driving path.

2. The method of claim 1 , wherein the first map comprises a probabilistic occupancy, by the EV, of a plurality of locations of the driving environment of the vehicle.

3. The method of claim 2 , wherein the SL output comprises a second map of possible velocities of the EV.

4. The method of claim 3 , wherein the second map comprises a plurality of probabilities, wherein each of the plurality of probabilities characterizes a likelihood that the EV is moving with a respective velocity of a plurality of velocities, and wherein simulating trajectories of the one or more simulated EVs comprises:

selecting, using the first map, locations for the one or more simulated EVs;

selecting, using the second map, velocities for the one or more simulated EVs; and

computing the simulated trajectories using the selected locations and the selected velocities.

5. The method of claim 1 , wherein applying the SL model to the sound recording comprises:

obtaining a first spectrogram for the sound recording, the first spectrogram obtained using a first sampling window size and a first sampling rate;

obtaining a second spectrogram for the sound recording, the second spectrogram obtained using a second sampling window size and a second sampling rate, wherein the first sampling window size is greater than the second sampling window size, and wherein the first sampling rate is smaller than the second sampling rate; and

processing, using the SL model, the first spectrogram and the second spectrogram.

6. The method of claim 5 , wherein processing the first spectrogram and the second spectrogram comprises:

processing, using a first neural network, the first spectrogram to obtain a first embedding;

processing, using a second neural network, the second spectrogram to obtain a second embedding;

fusing the first embedding and the second embedding to obtain a fused embedding; and

processing, using a third neural network, the fused embedding to obtain the SL output.

7. The method of claim 1 , wherein simulating trajectories of one or more simulated EVs comprises:

identifying, using electromagnetic sensor data collected by a sensing system of the vehicle, one or more occluded regions of the driving environment of the vehicle, wherein the electromagnetic sensor data comprises one or more of:

lidar data,

radar data, or

camera data; and

selecting locations of the one or more simulated EVs within the one or more occluded regions.

8. The method of claim 7 , further comprising:

identifying, using a road layout information, drivable areas within the one or more occluded regions; and

wherein the selected locations of the one or more simulated EVs are selected within the identified drivable areas.

9. The method of claim 7 , wherein the SL output further comprises a confidence score, the method further comprises:

responsive to the confidence score being less than a threshold confidence score, disregarding the first map in selecting the locations of the one or more simulated EVs.

10. The method of claim 1 , wherein the proximity of the one or more of the simulated trajectories to the driving path of the vehicle comprises a distance from the vehicle, projected to follow the driving path over a predetermined time, to the one or more of the simulated trajectories being less than a predetermined distance.

11. The method of claim 1 , wherein the SL model is trained using a plurality of training sound recordings obtained by a plurality of audio sensors, and wherein during at least one training epoch, one or more training sound recordings of the plurality of training sound recordings are replaced with null inputs into the SL model.

12. A system comprising:

a sensing system of a vehicle, the sensing system comprising one or more audio detectors configured to:

obtain a sound recording comprising a sound emitted by an emergency vehicle (EV); and

a perception system of the vehicle, the perception system configured to:

apply a sound localization (SL) model to the sound recording to obtain an SL output, wherein the SL output comprises a first map of possible locations of the EV in a driving environment of the vehicle;

simulate, using the SL output, trajectories of one or more simulated EVs in the driving environment of the vehicle;

cause, responsive to proximity of one or more of the simulated trajectories to a driving path of the vehicle, the driving path of the vehicle to be modified; and

control the vehicle based on the modified driving path.

13. The system of claim 12 , wherein the first map comprises a probabilistic occupancy, by the EV, of a plurality of locations of the driving environment of the vehicle, and wherein the SL output comprises a second map of possible velocities of the EV.

14. The system of claim 13 , wherein the second map comprises a plurality of probabilities, wherein each of the plurality of probabilities characterizes a likelihood that the EV is moving with a respective velocity of a plurality of velocities, and wherein to simulate trajectories of the one or more simulated EVs, the perception system is configured to:

select, using the first map, locations for the one or more simulated EVs;

select, using the second map, velocities for the one or more simulated EVs; and

compute the simulated trajectories using the selected locations and the selected velocities.

15. The system of claim 12 , wherein to apply the SL model to the sound recording, the perception system is to:

obtain a first spectrogram for the sound recording, the first spectrogram obtained using a first sampling window size and a first sampling rate;

obtain a second spectrogram for the sound recording, the second spectrogram obtained using a second sampling window size and a second sampling rate, wherein the first sampling window size is greater than the second sampling window size, and wherein the first sampling rate is smaller than the second sampling rate; and

process, using the SL model, the first spectrogram and the second spectrogram.

16. The system of claim 12 , wherein to simulate trajectories of one or more simulated EVs the perception system is to:

identify, using electromagnetic sensor data collected by a sensing system of the vehicle, one or more occluded regions of the driving environment of the vehicle, wherein the electromagnetic sensor data comprises one or more of:

lidar data,

radar data, or

camera data; and

select locations of the one or more simulated EVs within the one or more occluded regions.

17. The system of claim 16 , wherein the perception system is further configured to:

identify, using a road layout information, drivable areas within the one or more occluded regions; and

wherein the selected locations of the one or more simulated EVs are selected within the identified drivable areas.

18. The system of claim 16 , wherein the SL output further comprises a confidence score, and wherein the perception system is further to:

responsive to the confidence score being less than a threshold confidence score, disregarding the first map in selecting the locations of the one or more simulated EVs.

19. The system of claim 12 , wherein the SL model is trained using a plurality of training sound recordings obtained by a plurality of audio sensors, and wherein during at least one training epoch, one or more training sound recordings of the plurality of training sound recordings are replaced with null inputs into the SL model.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a processing device cause the processing device to perform operations comprising:

obtaining, using one or more audio detectors of a vehicle, a sound recording comprising a sound emitted by an emergency vehicle (EV);

applying a sound localization (SL) model to the sound recording to generate an SL output, wherein the SL output comprises a first map of possible locations of the EV in a driving environment of the vehicle;

simulating, using the SL output, trajectories of one or more simulated EVs in the driving environment of the vehicle;

causing, responsive to proximity of one or more of the simulated trajectories to a driving path of the vehicle, the driving path of the vehicle to be modified; and

controlling the vehicle based on the modified driving path.