Object tracking for autonomous vehicles using long-range acoustic beamforming combined with RGB visual data
An autonomous vehicle including a network of sensors including a plurality of acoustic sensors and a plurality of visual sensors, at least one processor, and at least one memory storing instructions is disclosed. The instructions, when executed by the at least one processor, cause the at least one processor to: (i) generate spatial beamforming maps locating a sound source based upon acoustic signals received at the plurality of acoustic sensors; (ii) identify a type of an object generating the acoustic signals received at the plurality of acoustic sensors based upon comparison of the acoustic signals with a plurality of acoustic signals and respective objects stored in a dataset; and (iii) generate feature maps for an application in an autonomous vehicle driving by enhancing visualization maps generated based upon visual signals received by the plurality of visual sensors.
1 . An autonomous vehicle, comprising:
a network of sensors including a plurality of acoustic sensors and a plurality of visual sensors;
at least one processor; and
at least one memory storing instructions, which, when executed by the at least one processor, cause the at least one processor to:
generate spatial beamforming maps locating a sound source based upon acoustic signals received at the plurality of acoustic sensors;
identify a type of an object generating the acoustic signals received at the plurality of acoustic sensors based upon comparison of the acoustic signals with a plurality of acoustic signals and respective objects stored in a dataset, wherein the dataset is a multimodal long-range beamforming dataset including audio visual data corresponding to a plurality of objects; and
generate feature maps for an application in an autonomous vehicle driving by enhancing visualization maps generated based upon visual signals received by the plurality of visual sensors.
2 . The autonomous vehicle of claim 1 , wherein the application in the autonomous vehicle driving is an object detection application or a future RGB frame detection application for planning and behavior control of the autonomous vehicle.
3 . The autonomous vehicle of claim 1 , wherein the plurality of visual sensors includes one or more RGB cameras, one or more serial cameras, or one or more lidar sensors.
4 . The autonomous vehicle of claim 1 , wherein the plurality of acoustic sensors is arranged in a grid pattern.
5 . The autonomous vehicle of claim 4 , wherein a grid spacing between the plurality of acoustic sensors in the grid pattern is selected based on upper frequency bounds of the acoustic signals.
6 . The autonomous vehicle of claim 1 , wherein the audio visual data is synchronized with a global navigation satellite system as a time reference.
7 . A computer-implemented method, comprising:
generating spatial beamforming maps locating a sound source based upon acoustic signals received at a plurality of acoustic sensors of a network of sensors;
identifying a type of an object generating the acoustic signals received at the plurality of acoustic sensors based upon comparison of the acoustic signals with a plurality of acoustic signals and respective objects stored in a dataset, wherein the dataset is a multimodal long-range beamforming dataset including audio visual data corresponding to a plurality of objects; and
generating feature maps for an application in an autonomous vehicle driving by enhancing visualization maps generated based upon visual signals received by a plurality of visual sensors of the network of sensors.
8 . The computer-implemented method of claim 7 , wherein the application in the autonomous vehicle driving is an object detection application or a future RGB frame detection application for planning and behavior control of the autonomous vehicle.
9 . The computer-implemented method of claim 7 , wherein the plurality of visual sensors includes one or more RGB cameras, one or more serial cameras, or one or more lidar sensors.
10 . The computer-implemented method of claim 7 , wherein the plurality of acoustic sensors is arranged in a grid pattern.
11 . The computer-implemented method of claim 10 , wherein a grid spacing between the plurality of acoustic sensors in the grid pattern is selected based on upper frequency bounds of the acoustic signals.
12 . The computer-implemented method of claim 7 , wherein the audio visual data is synchronized with a global navigation satellite system as a time reference.
13 . A non-transitory computer-readable medium (CRM) embodying programmed instructions which, when executed by at least one processor of an autonomous vehicle, cause the at least one processor to perform operations comprising:
generating spatial beamforming maps locating a sound source based upon acoustic signals received at a plurality of acoustic sensors of a network of sensors;
identifying a type of an object generating the acoustic signals received at the plurality of acoustic sensors based upon comparison of the acoustic signals with a plurality of acoustic signals and respective objects stored in a dataset, wherein the dataset is a multimodal long-range beamforming dataset including audio visual data corresponding to a plurality of objects; and
generating feature maps for an application in an autonomous vehicle driving by enhancing visualization maps generated based upon visual signals received by a plurality of visual sensors of the network of sensors.
14 . The non-transitory CRM of claim 13 , wherein the application in the autonomous vehicle driving is an object detection application or a future RGB frame detection application for planning and behavior control of the autonomous vehicle.
15 . The non-transitory CRM of claim 13 , wherein the plurality of visual sensors includes one or more RGB cameras, one or more serial cameras, or one or more lidar sensors.
16 . The non-transitory CRM of claim 13 , wherein the plurality of acoustic sensors is arranged in a grid pattern.
17 . The non-transitory CRM of claim 16 , wherein a grid spacing between the plurality of acoustic sensors in the grid pattern is selected based on upper frequency bounds of the acoustic signals.
18 . The non-transitory CRM of claim 13 , wherein the audio visual data is synchronized with a global navigation satellite system as a time reference.