Multi-sensor fusion platform for bootstrapping the training of a beam steering radar
Examples disclosed herein relate to a multi-sensor fusion platform for use in autonomous vehicles, the multi-sensor fusion platform including a camera perception engine having a camera neural network to detect and identify objects in camera data, a lidar perception engine having a lidar neural network to detect and identify objects in lidar data, and a radar perception engine having a radar neural network to detect and identify objects in radar data, such that training of the radar neural network is bootstrapped with the camera and lidar neural networks.
1. A radar system, comprising:
a camera perception engine comprising a camera neural network, the camera perception engine configured to detect and identify one or more objects in camera data;
a lidar perception engine comprising a lidar neural network, the lidar perception engine configured to detect and identify one or more objects in lidar data;
a system controller for controlling the radar system; and
a radar perception engine comprising a radar neural network, the radar perception engine coupled to a beam steering radar and configured to detect and identify one or more objects in radar data from the beam steering radar,
wherein the system controller is adapted for training of the radar neural network using training of the camera perception engine and the lidar perception engine.
2. The radar system of claim 1 , wherein training of the radar perception engine uses information from training of the camera and lidar neural networks.
3. The radar system of claim 1 , wherein information from the camera perception engine and the lidar perception engine augment object detection and identification capabilities of the radar precepting engine.
4. The radar system of claim 1 , wherein the radar perception engine is further configured to adjust scanning parameters of the beam steering radar to avoid or minimize interference from one or more interfering radar signals that is determined to be interfering with an autonomous vehicle.
5. The radar system of claim 4 , wherein the autonomous vehicle comprises one or more cameras, one or more lidars, one or more infrastructure sensors, one or more environmental sensors, one or more operational sensors, or one or more user preference sensors.
6. The radar system of claim 5 , wherein the one or more environmental sensors is configured to detect a weather condition or a visibility condition of an environment of the autonomous vehicle.
7. The radar system of claim 6 , wherein the one or more environmental sensors is further configured to be turned off or reduce its sampling rate based on the detected weather condition or the detected visibility condition.
8. The radar system of claim 7 , wherein the beam steering radar reduces a width of a radar beam transmitted to provide a more focused beam for a finer sensing capability.
9. The radar system of claim 1 , wherein the training of the radar neural network includes annotating the radar data with bounded boxes and identifying and categorizing the one or more detected objects based on the annotating.
10. The radar system of claim 1 , wherein the one or more detected and identified objects based on the camera data, the lidar data, and the radar data are input into a Kalman filter for tracking over time.
11. An object detection and classification method for a radar system, comprising:
generating object detection labels by annotating data from a plurality of camera, lidar, and radar sensors;
accessing a radar neural network trained with the object detection labels, the radar neural network comprising a Kalman filter;
generating, via the Kalman filter, updated object detection labels; and
retraining the radar neural network based on the updated object detection labels.
12. The method of claim 11 , wherein the object detection labels comprise bounded boxes that enable identification and categorization of one or more detected objects.
13. The method of claim 11 , wherein the radar neural network is further trained based on trained camera and lidar neural networks.
14. The method of claim 11 , wherein the generation of the updated object detection labels via the Kalman filter includes matching candidate objects detected by one or more of the plurality of camera, lidar, and radar sensors with objects that have been detected in previous time windows.
15. The method of claim 11 , further comprising:
regenerating the updated object detection labels based on the retrained radar neural network.
16. The method of claim 15 , further comprising:
retraining the radar neural network based on the regenerated updated object detection labels.
17. A method for object detection and classification in a radar system, comprising:
acquiring data via a plurality of camera and lidar sensors and a beam steering radar;
generating super-resolution and object detection labels by annotating the acquired data from the plurality of camera and lidar sensors and the beam steering radar;
training a radar neural network with the super-resolution and object detection labels, wherein the radar neural network is coupled to the beam steering radar; and
regenerating the object detection labels at a Kalman filter coupled to the radar neural network.
18. The method of claim 17 , further comprising increasing a resolution of radar data for human annotators using the super-resolution labels in a super-resolution network.
19. The method of claim 17 , wherein training the radar neural network is based on information from trained camera and lidar neural networks.
20. The method of claim 17 , wherein regenerating of the object detection labels at the Kalman filter includes matching candidate objects detected in the acquired data with objects that have been detected in previous time windows.