Label transfer between data from multiple sensors
A method includes receiving first sensor data captured by a first sensor. The method further includes receiving a plurality of labels or predictions corresponding to the first sensor data. The method also includes receiving second sensor data captured by a second sensor. The method further includes determining time-synchronized sensor data comprising a subset of the first sensor data and a subset of the second sensor data. The method additionally includes determining, based on the plurality of labels or predictions and the time-synchronized sensor data, a plurality of pseudo-labels corresponding to the second sensor data. The method also includes generating a training data set comprising at least the subset of the second sensor data and one or more pseudo-labels from the plurality of pseudo-labels.
1. A method comprising:
receiving first sensor data captured by a first sensor, wherein the first sensor data includes two-dimensional image data and depth data;
receiving a plurality of labels or predictions defining coordinates of one or more two-dimensional bounding boxes of the first sensor data;
receiving second sensor data captured by a second sensor, wherein the second sensor data includes point cloud data;
determining time-synchronized sensor data comprising a subset of the first sensor data and a subset of the second sensor data;
determining, based on the plurality of labels or predictions, the time-synchronized sensor data, and the depth data, a plurality of pseudo-labels corresponding to the second sensor data, wherein the plurality of pseudo-labels define coordinates of one or more three-dimensional bounding boxes based on the one or more two-dimensional bounding boxes and the depth data; and
generating a training data set comprising at least the subset of the second sensor data and one or more pseudo-labels from the plurality of pseudo-labels.
2. The method of claim 1 , wherein the first sensor is a red green blue depth (RGB-D) camera, wherein the second sensor is a LIDAR sensor.
3. The method of claim 1 , wherein the second sensor has a wider field of view than the first sensor.
4. The method of claim 1 , wherein the first sensor data represents an area smaller than the second sensor data, and wherein determining the plurality of pseudo-labels corresponding to the second sensor data comprises:
determining, based on the first sensor and the second sensor, cropped second sensor data corresponding to the area represented by the first sensor data.
5. The method of claim 1 , wherein the plurality of labels or predictions comprise labels determined by using a human-assisted labeling process.
6. The method of claim 1 , wherein the plurality of labels or predictions comprise predictions determined by using an automated prediction process run on a computing device.
7. The method of claim 1 , wherein the plurality of predictions or labels indicate one or more object locations in the first sensor data, and wherein the plurality of pseudo-labels indicate one or more object locations in the second sensor data.
8. The method of claim 1 , wherein the time-synchronized sensor data comprising a subset of the first sensor data and a subset of the second sensor data is based on a threshold timestamp difference between the first sensor data and the second sensor data.
9. The method of claim 1 , wherein the subset the first sensor data is an improper subset of the first sensor data.
10. The method of claim 1 , wherein the subset of the second sensor data is an improper subset of the second sensor data.
11. The method of claim 1 , wherein the method further comprises before generating the training data set, validating, based at least on the subset of the second sensor data and a quality measure associated with the plurality of pseudo-labels corresponding to the second sensor data, one or more pseudo-labels from the plurality of pseudo-labels and wherein the training data set is generated based on the one or more validated pseudo-labels.
12. The method of claim 11 , wherein the second sensor is a LIDAR sensor, wherein the second sensor data includes point cloud data, and wherein the quality measure is a threshold number of points within the point cloud data.
13. The method of claim 1 , wherein the method further comprises determining, based at least on the plurality of pseudo-labels, confidence scores, wherein generating the training data set is based on the confidence scores.
14. The method of claim 13 , wherein the method further comprises:
determining, based on the confidence scores, a high confidence subset of the second sensor data and a high confidence subset of the plurality of pseudo-labels; and
generating a high confidence training data set comprising at least the high confidence subset of the second sensor data and the high confidence subset of the plurality of pseudo-labels, wherein the high confidence training data set is to be used in supervised machine learning.
15. A system comprising:
a first sensor;
a second sensor; and
a computing device configured to:
receive first sensor data captured by a first sensor, wherein the first sensor data includes two-dimensional image data and depth data;
receive a plurality of labels or predictions defining coordinates of one or more two-dimensional bounding boxes of the first sensor data;
receive second sensor data captured by a second sensor, wherein the second sensor data includes point cloud data;
determine time-synchronized sensor data comprising a subset of the first sensor data and a subset of the second sensor data;
determine, based on the plurality of labels or predictions, the time-synchronized sensor data, and the depth data, a plurality of pseudo-labels corresponding to the second sensor data, wherein the plurality of pseudo-labels define coordinates of one or more three-dimensional bounding boxes based on the one or more two-dimensional bounding boxes and the depth data; and
generate a training data set comprising at least the subset of the second sensor data and one or more pseudo-labels from the plurality of pseudo-labels.
16. The system of claim 15 , wherein the computing device is further configured to:
receive additional second sensor data captured by the second sensor;
receive a second plurality of predictions corresponding to the additional second sensor data, wherein the second plurality of predictions was generated by a machine learning model trained on the plurality of pseudo-labels, wherein the second plurality of predictions were generated by applying the machine learning model to the additional second sensor data;
receive additional first sensor data captured by the first sensor;
determine additional time-synchronized sensor data comprising a subset of the additional second sensor data and a subset of the additional first sensor data;
determine, based on the second plurality of predictions and the additional time-synchronized sensor data, an additional plurality of pseudo-labels corresponding to the additional first sensor data; and
generate an additional training data set comprising at least the subset of the additional first sensor data and an additional one or more pseudo-labels from the additional plurality of pseudo-labels.
17. A non-transitory computer readable medium comprising program instructions executable by at least one processor to cause the at least one processor to perform functions comprising:
receiving first sensor data captured by a first sensor, wherein the first sensor data includes two-dimensional image data and depth data;
receiving a plurality of labels or predictions defining coordinate of one or more two-dimensional bounding boxes of the first sensor data;
receiving second sensor data captured by a second sensor, wherein the second sensor data includes point cloud data;
determining time-synchronized sensor data comprising a subset of the first sensor data and a subset of the second sensor data;
determining, based on the plurality of labels or predictions, the time-synchronized sensor data and the depth data, a plurality of pseudo-labels corresponding to the second sensor data, wherein the plurality of pseudo-labels define coordinates of one or more three-dimensional bounding boxes based on the one or more two-dimensional bounding boxes and the depth data; and
generating a training data set comprising at least the subset of the second sensor data and one or more pseudo-labels from the plurality of pseudo-labels.
18. The method of claim 1 , wherein the one or more two-dimensional bounding boxes are two-dimensional rectangles, and the one or more three-dimensional bounding boxes are three-dimensional rectangles.