Representation learning for object detection from unlabeled point cloud sequences
A method of representation learning for object detection from unlabeled point cloud sequences is described. The method includes detecting moving object traces from temporally-ordered, unlabeled point cloud sequences. The method also includes extracting a set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences. The method further includes classifying the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences. The method also includes estimating 3D bounding boxes for the set of moving objects based on the classifying of the set of moving objects.
1. A method of representation learning for object detection from unlabeled point cloud sequences, comprising:
detecting moving object traces from temporally-ordered, unlabeled point cloud sequences;
extracting a set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences;
classifying the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences;
estimating 3D bounding boxes for the set of moving objects based on the classifying of the set of moving objects;
labeling the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences as moving vehicles; and
planning a trajectory of an ego vehicle according to the labeled moving vehicles in a scene surrounding the ego vehicle.
2. The method of claim 1 , in which the detecting moving object traces comprises:
visualizing the temporally-ordered, unlabeled point cloud sequences in a world coordinate system according to an ego-motion;
removing ground points from the visualizing the temporally-ordered, unlabeled point cloud sequences to form a ground removed point cloud visualization;
estimating object cluster proposals according to a point cloud segmentation of the ground removed point cloud visualization; and
identifying the moving object traces from the object cluster proposal according to multi-object tracing.
3. The method of claim 1 , in which extracting the set of moving objects comprises:
training a single-frame semantic instance segmentation model to differentiate between feature vectors representing the set of moving objects and the feature vectors representing a background of the temporally-ordered, unlabeled point cloud sequences; and
identifying each of the feature vectors as a moving feature vector or a non-moving feature vector.
4. The method of claim 1 , further comprises training a feature extraction module to extract the set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point clouds via self-supervised tasks.
5. The method of claim 1 , in which the set of moving objects are represented as a sequence of point clusters that correspond to corresponding one of the set of moving objects.
6. The method of claim 1 , further comprising:
training a first model to identify moving objects in a point cloud according to the labeled moving object traces;
inferring attributes of the bounding boxes from the labeled moving object traces; and
training a second model to detect objects in the point cloud according to the attributes of the bounding boxes inferred from the labeled moving object traces.
7. The method of claim 1 , in which estimating the 3D bounding boxes comprises:
registering the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences according to object class labels; and
assigning the 3D bounding boxes according to the object class labels.
8. The method of claim 1 , further comprising controlling the ego vehicle along the planned trajectory of the ego vehicle according to the labeled moving vehicles detected in the scene surrounding the ego vehicle.
9. A non-transitory computer-readable medium having program code recorded thereon for representation learning and object detection from unlabeled point cloud sequences, the program code being executed by a processor and comprising:
program code to detect moving object traces from temporally-ordered, unlabeled point cloud sequences;
program code to extract a set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences;
program code to classify the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences;
program code to estimate 3D bounding boxes for the set of moving objects based on the classifying of the set of moving objects;
program code to label the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences as moving vehicles; and
program code to plan a trajectory of an ego vehicle according to the labeled moving vehicles in a scene surrounding the ego vehicle.
10. The non-transitory computer-readable medium of claim 9 , in which the program code to detect moving object traces comprises:
program code to visualize the temporally-ordered, unlabeled point cloud sequences in a world coordinate system according to an ego-motion;
program code to remove ground points from the visualizing the temporally-ordered, unlabeled point cloud sequences to form a ground removed point cloud visualization;
program code to estimate object cluster proposals according to a point cloud segmentation of the ground removed point cloud visualization; and
program code to identify the moving object traces from the estimated object cluster proposal according to multi-object tracing.
11. The non-transitory computer-readable medium of claim 9 , in which the program code to extract the set of moving objects comprises:
program code to train a single-frame semantic instance segmentation model to differentiate between feature vectors representing the set of moving objects and the feature vectors representing a background of the temporally-ordered, unlabeled point cloud sequences; and
program code to identify each of the feature vectors as a moving feature vector or a non-moving feature vector.
12. The non-transitory computer-readable medium of claim 9 , further comprises program code to train a feature extraction module to extract the set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point clouds via self-supervised tasks.
13. The non-transitory computer-readable medium of claim 9 , in which the set of moving objects are represented as a sequence of point clusters that correspond to corresponding one of the set of moving objects.
14. The non-transitory computer-readable medium of claim 9 , further comprising:
program code to train a first model to identify moving objects in a point cloud according to the labeled moving object traces;
program code to infer attributes of the bounding boxes from the labeled moving object traces; and
program code to train a second model to detect objects in the point cloud according to the attributes of the bounding boxes inferred from the labeled moving object traces.
15. The non-transitory computer-readable medium of claim 9 , in which the program code to estimate the 3D bounding boxes comprises:
program code to register the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences according to object class labels; and
program code to assign the 3D bounding boxes according to the object class labels.
16. The non-transitory computer-readable medium of claim 9 , further comprising program code to control the ego vehicle along the planned trajectory of the ego vehicle according to the labeled moving vehicles detected in the scene surrounding the ego vehicle.
17. A system of representation learning for object detection from unlabeled point cloud sequences, the system comprising:
a moving object trace detection module to detect moving object traces from temporally-ordered, unlabeled point cloud sequences;
a moving object extraction module to extract a set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences;
an object classification and labeling module to classify the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences;
a bounding box estimation module to estimate 3D bounding boxes for the set of moving objects based on the classifying of the set of moving objects, and to label the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences as moving vehicles; and
a planner to plan a trajectory of an ego vehicle according to the labeled moving vehicles in a scene surrounding the ego vehicle.
18. The system of claim 17 , further comprising a single-frame semantic instance segmentation model trained to differentiate between feature vectors representing the set of moving objects and the feature vectors representing a background of the temporally-ordered, unlabeled point cloud sequences and to identify each of the feature vectors as a moving feature vector or a non-moving feature vector.
19. The system of claim 17 , in which the set of moving objects are represented as a sequence of point clusters that correspond to corresponding one of the set of moving objects.
20. The system of claim 17 , further comprising a controller to control the ego vehicle along the planner trajectory of the ego vehicle according to the 3D bounding boxes detected in the scene surrounding the ego vehicle.