Systems and methods for camera-to-radar knowledge distillation
A workflow is described herein for training a radar data processing model with a reduced requirement for annotated training data. The training workflow leverages an image sensor, such as a camera, that is synchronized with a radar sensor to capture synchronous image and radar point cloud data. The training workflow utilizes a self-supervised knowledge distillation process to pre-train the radar data processing model on the unannotated synchronous image and radar point cloud data, using a pre-trained image model. Subsequently, the radar data processing model is fine-tuned with a limited set of annotated radar point cloud data, thereby greatly reducing the human annotation burden.
1 . A method for training a first neural network to perform a radar data processing task, the method comprising:
receiving, with a processor, a plurality of training data pairs, each respective training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud, which were captured synchronously with one another of a same scene;
training, with the processor, in a first phase based on the plurality of training data pairs, the first neural network to extract features from radar point clouds using a second neural network that is pre-trained to extract features from images;
receiving, with the processor, a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to a radar data processing task; and
further training, with the processor, in a second phase based on the plurality of annotated radar point clouds, the first neural network to perform the radar data processing task.
2 . The method according to claim 1 , wherein:
the first neural network comprises an encoder followed by a decoder,
the training the first neural network in the first phase comprises training the encoder of the first neural network using the second neural network and based on the plurality of training data pairs; and
the training the first neural network in the second phase comprises training the decoder of the first neural network based on the plurality of annotated radar point clouds.
3 . The method according to claim 2 , wherein parameters of the encoder of the first neural network are frozen during training the decoder of the first neural network in the second phase.
4 . The method according to claim 1 , wherein parameters of the second neural network are frozen during the training the first neural network in the first phase.
5 . The method according to claim 1 , the training the first neural network in the first phase comprising:
determining, using the first neural network, a first feature output based on a respective radar point cloud in the plurality of training data pairs;
determining, using the second neural network, a second feature output based on a respective image in the plurality of training data pairs that corresponds to the respective radar point cloud; and
refining the first neural network based on the first feature output and the second feature output.
6 . The method according to claim 5 , the training the first neural network in the first phase comprising:
determining a mapping between (i) points of radar point clouds in the plurality of training data pairs and (ii) pixels of images in the plurality of training data pairs.
7 . The method according to claim 6 , the training the first neural network in the first phase comprising:
determining a contrastive loss based on the first feature output and the second feature output and based on the mapping; and
refining the first neural network based on the contrastive loss.
8 . The method according to claim 7 , the training the first neural network in the first phase comprising:
segmenting the respective radar point cloud into superpoints, each superpoint including a subset of points from the respective radar point cloud; and
segmenting the respective image into superpixels, each superpixel including a subset of pixels from the respective image.
9 . The method according to claim 8 , the training the first neural network in the first phase comprising:
determining, for each respective superpoint in the respective radar point cloud, a respective superpoint feature output;
determining, for each respective superpixel in the respective image, a respective superpixel feature output; and
determining the contrastive loss based on the respective superpoint feature output of each superpoint in the respective radar point cloud and the respective superpixel feature output of each superpixel in the respective image.
10 . The method according to claim 9 , the training the first neural network in the first phase comprising:
matching each superpoint in the respective radar point cloud to a respective superpixel in the respective image; and
determining the contrastive loss based on the respective superpoint feature output and the respective superpixel feature output of each matched superpoint and superpixel.
11 . The method according to claim 9 , the determining the respective superpoint feature output further comprising:
determining the respective superpoint feature output as a weighted average of features in the first feature output for points of the respective superpoint, the average being weighted depending on an average distance of each point with each other point in the respective superpoint.
12 . The method according to claim 9 , the determining the respective superpixel feature output further comprising:
determining the respective superpixel feature output as an average of features in the second feature output for pixels of the respective superpixel.
13 . The method according to claim 8 , the segmenting the respective image into superpixels further comprising:
defining each superpixel of the respective image by applying at least one of an image segmentation algorithm and a pixel clustering algorithm to the respective image.
14 . The method according to claim 13 , the segmenting the respective radar point cloud into superpoints further comprising:
matching each point in the respective radar point cloud to a respective superpixel in the respective image; and
defining each superpoint of the respective radar point cloud as a group of radar points that match to a same superpixel in the respective image.
15 . The method according to claim 8 , the segmenting the respective radar point cloud into superpoints further comprising:
defining each superpoint of the respective radar point cloud by applying a point clustering algorithm to the respective radar point cloud.
16 . The method according to claim 15 , the segmenting the respective image into superpixels further comprising:
matching each superpixel in the respective image to a respective superpoint in the respective radar point cloud; and
defining each superpixel of the respective image as a group of pixels that match to a same superpoint in the respective radar point cloud.
17 . The method according to claim 15 , the segmenting the respective image into superpixels further comprising:
initially defining superpixels of the respective image by applying at least one of an image segmentation algorithm and a pixel clustering algorithm to the respective image;
matching each initially defined superpixel in the respective image to a respective superpoint in the respective radar point cloud; and
defining each superpixel of the respective image as a group of initially defined superpixels that match to a same superpoint in the respective radar point cloud.
18 . The method according to claim 15 , the applying the point clustering algorithm to the respective radar point cloud further comprising:
generating a respective combined radar point cloud by combining the respective radar point cloud with at least one further radar point cloud that was captured at an immediately previous or subsequent time compared to a time at which the respective radar point cloud was captured.
19 . The method according to claim 18 , the applying the point clustering algorithm to the respective radar point cloud further comprising:
defining each superpoint of the respective radar point cloud by applying a point clustering algorithm to the respective combined radar point cloud.
20 . A non-transitory computer-readable medium for training a first neural network to perform a radar data processing task, the non-transitory computer-readable medium storing program instructions that, when executed by a processor, cause the processor to:
receive a plurality of training data pairs, each respective training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud, which were captured synchronously with one another of a same scene;
train, in a first phase based on the plurality of training data pairs, the first neural network to extract features from radar point clouds using a second neural network that is pre-trained to extract features from images;
receive a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to a radar data processing task; and
further train, in a second phase based on the plurality of annotated radar point clouds, the first neural network to perform the radar data processing task.