Granted Patent B2

US 12,437,554 · App. 18/170,642 · Granted Oct 7, 2025

Inventors: Arindam Das (Mumbai, IN); Sudarshan Paul (Mumbai, IN); Sanjoy Das (Mumbai, IN); Deep Doshi (Troy, MI)

Assignee: Connaught Electronics Ltd.

G06V20/58G06N3/0464B60W2420/403B60W2420/54B60W2422/95

Abstract

Claims (82)

1. A computer-implemented method for training a perception algorithm of a vehicle, the computer-implemented method comprising:

providing a convolutional recurrent neural network (CRNN) and at least one further artificial neural network (ANN) to detect at least one emergency vehicle;

receiving at least two time-dependent audio datasets from two microphones mounted at different positions on the vehicle;

generating at least two spectrograms based on the at least two time-dependent audio datasets;

generating at least one interaural difference map based on the at least two spectrograms, which contains at least one of an interaural phase difference map or an interaural time difference map or an interaural level difference map;

generating audio source localization data for at least one grid cell of a predefined spatial grid in an environment of the vehicle;

specifying a number of audio sources in the at least one grid cell;

applying the CRNN to a first input data containing the at least two spectrograms and the at least one interaural difference map;

receiving at least one camera image from at least one camera mounted to the vehicle;

predicting output data of at least one bounding box for the at least one emergency vehicle by applying the at least one further ANN to a second input data containing the at least one camera image and the at least two spectrograms; and

adapting network parameters of the CRNN and the at least one further ANN by depending on the output data and the audio source localization data.

2. The computer-implemented method of claim 1 , wherein adapting the network parameters further comprising:

providing a ground truth for the at least one bounding box given by or dependent upon the audio source localization data.

3. The computer-implemented method of claim 1 , further comprising:

generating at least one visual domain feature map by applying a first object detection ANN of the at least one further ANN to the at least one camera image;

generating at least one audio domain feature map by applying an audio source localization ANN of the at least one further ANN to the at least two spectrograms;

fusing the at least one visual domain feature map and the at least one audio domain feature map; and

predicting the output data by applying a second object detection ANN of the at least one further ANN to the fused feature maps.

4. The computer-implemented method of claim 3 , wherein fusing the at least one visual domain feature map and the at least one audio domain feature map further comprising:

generating at least one difference map by subtracting the at least one visual domain feature map and the at least one audio domain feature map from each other;

generating at least one re-calibration vector based on the at least one difference map;

multiplying the at least one visual domain feature map with the at least one re-calibration vector to generate at least one re-calibrated visual domain feature map;

multiplying the at least one audio domain feature map with the at least one re-calibration vector to generate at least one re-calibrated audio domain feature map; and

fusing the at least one re-calibrated video domain feature map and the at least one re-calibrated audio domain feature map.

5. The computer-implemented method of claim 4 , wherein generating the at least one re-calibration vector further comprising:

applying sum pooling, followed by a normalization, followed by a global average pooling, followed by an activation function, to the at least one difference map.

6. The computer-implemented method of claim 3 , wherein the first object detection ANN is designed for object detection according to a predefined set of object classes,

wherein the set of object classes contains a class for activated emergency vehicle lights.

7. The computer-implemented method of claim 1 , wherein each audio dataset of the at least two time-dependent audio datasets further comprising:

generating a corresponding two-dimensional single magnitude map as a function of time and audio amplitude,

wherein the first input data comprises the single magnitude maps.

8. A perception system for a vehicle, the perception system comprising:

a data processing apparatus;

a memory device configured to store a perception algorithm;

a convolutional recurrent neural network (CRNN) and at least one further artificial neural network (ANN) configured to detect at least one emergency vehicle;

at least two microphones mounted at different positions on the vehicle configured to generate at least two time-dependent audio datasets;

at least one camera mounted to the vehicle configured to generate at least one camera image; and

a computing unit configured to:

generate at least two spectrograms based on the at least two time-dependent audio datasets;

generate at least one interaural difference map based on the at least two spectrograms, which contains at least one of an interaural phase difference map or an interaural time difference map or an interaural level difference map:

generate audio source localization data for at least one grid cell of a predefined spatial grid in an environment of the vehicle;

specify a number of audio sources in the at least one grid cell:

apply the CRNN to a first input data containing the at least two spectrograms and the at least one interaural difference map;

predict output data of at least one bounding box for the at least one emergency vehicle by applying the at least one further ANN to a second input data containing the at least one camera image and the at least two spectrograms; and

adapt network parameters of the CRNN and the at least one further ANN by depending on the output data and the audio source localization data.

9. The perception system of claim 8 , wherein the at least one camera image depicts the environment of the vehicle, and

wherein the computing unit is configured to predict output data including at least one bounding box for at least one emergency vehicle in the environment of the vehicle by applying the at least one further ANN to the second input data containing the at least one camera image and the at least two spectrograms.

10. The perception system of claim 8 , wherein the at least two microphones are implemented as respective uni-directional microphones.

11. The perception system of claim 8 , wherein the at least two microphones includes a first microphone mounted in a front half of the vehicle and a second microphone is mounted in a rear half of the vehicle.

12. The perception system of claim 8 , wherein the at least two microphones are configured to have a respective direction of maximum sensitivity parallel to a longitudinal axis of the vehicle.

13. The perception system of claim 8 , wherein the data processing apparatus is configured to execute at least one instructions.

14. The perception system of claim 8 , wherein adapt the network parameters further comprises:

provide a ground truth for the at least one bounding box given by or dependent upon the audio source localization data.

15. The perception system of claim 8 , wherein the computing unit further comprises:

generate at least one visual domain feature map by applying a first object detection ANN of the at least one further ANN to the at least one camera image;

generate at least one audio domain feature map by applying an audio source localization ANN of the at least one further ANN to the at least two spectrograms;

fuse the at least one visual domain feature map and the at least one audio domain feature map; and

predict the output data by applying a second object detection ANN of the at least one further ANN to the fused feature maps.

16. The perception system of claim 15 , wherein fuse the at least one visual domain feature map and the at least one audio domain feature map further comprises:

generate at least one difference map by subtracting the at least one visual domain feature map and the at least one audio domain feature map from each other;

generate at least one re-calibration vector based on the at least one difference map;

multiply the at least one visual domain feature map with the at least one re-calibration vector to generate at least one re-calibrated visual domain feature map;

multiply the at least one audio domain feature map with the at least one re-calibration vector to generate at least one re-calibrated audio domain feature map; and

fuse the at least one re-calibrated video domain feature map and the at least one re-calibrated audio domain feature map.

17. The perception system of claim 16 , wherein generate the at least one re-calibration vector further comprises:

apply sum pooling, followed by a normalization, followed by a global average pooling, followed by an activation function, to the at least one difference map.

18. The perception system of claim 15 , wherein the first object detection ANN is designed for object detection according to a predefined set of object classes,

wherein the set of object classes contains a class for activated emergency vehicle lights.

19. The perception system of claim 8 , wherein each audio dataset of the at least two time-dependent audio datasets further comprises:

generate a corresponding two-dimensional single magnitude map as a function of time and audio amplitude,

wherein the first input data comprises the single magnitude maps.

20. A non-transitory computer-readable storage medium storing instructions, which when executed on a computer, cause the computer to perform a method for training a perception algorithm of a vehicle, the method comprising:

providing a convolutional recurrent neural network (CRNN) and at least one further artificial neural network (ANN) to detect at least one emergency vehicle;

receiving at least two time-dependent audio datasets from two microphones mounted at different positions on the vehicle;

generating at least two spectrograms based on the at least two time-dependent audio datasets;

generating at least one interaural difference map based on the at least two spectrograms, which contains at least one of an interaural phase difference map or an interaural time difference map or an interaural level difference map;

generating audio source localization data for at least one grid cell of a predefined spatial grid in an environment of the vehicle;

specifying a number of audio sources in the at least one grid cell;

applying the CRNN to a first input data containing the at least two spectrograms and the at least one interaural difference map;

receiving at least one camera image from at least one camera mounted to the vehicle;

predicting output data of at least one bounding box for the at least one emergency vehicle by applying the at least one further ANN to a second input data containing the at least one camera image and the at least two spectrograms; and

adapting network parameters of the CRNN and the at least one further ANN by depending on the output data and the audio source localization data.

Continuity (1)

References Cited (12)

Cited By (1)

US 12,738,160