Convolutional neural network system for object detection and lane detection in a motor vehicle
View Patent ↗A system and method for predicting object detection and lane detection for a motor vehicle includes a convolution neural network (CNN) that receives an input image and a lane line module. The CNN includes a set of convolution and pooling layers (CPL's) trained to detect objects and lane markings from the input image, the objects categorized into object classes and the lane markings categorized into lane marking classes to generate a number of feature maps, a fully connected layer that receives the feature maps, the fully connected layer generating multiple object bounding box predictions for each of the object classes and multiple lane bounding box predictions for each of the lane marking classes from the feature maps, and a non-maximum suppression layer generating a final object bounding box prediction for each of the object classes and generating multiple final lane bounding box predictions for each of the lane marking classes.
1. A system for predicting object detection and lane detection for a motor vehicle, the system comprising:
a convolution neural network (CNN) receiving an input image, the CNN having:
a set of convolution and pooling layers (CPL's) trained to detect objects and lane markings from the input image, the objects categorized into object classes and the lane markings categorized into lane marking classes to generate a number of feature maps;
a fully connected layer that receives the feature maps, the fully connected layer generating multiple object bounding box predictions for each of the object classes and multiple lane bounding box predictions for each of the lane marking classes from the feature maps;
a non-maximum suppression layer receiving the multiple object bounding box predictions and the multiple lane bounding box predictions and generating a final object bounding box prediction for each of the object classes and generating multiple final lane bounding box predictions for each of the lane marking classes; and
a lane line module configured to generate lane lines from the multiple final lane bounding box predictions.
2. The system of claim 1 , wherein the lane marking classes include a left lane marking class and a right lane marking class.
3. The system of claim 2 , wherein the multiple final lane bounding box predictions are centered on the lane markings.
4. The system of claim 2 , wherein the multiple final lane bounding box predictions each include a confidence value.
5. The system of claim 2 , wherein the multiple final lane bounding box predictions each include localization information that locate the multiple final lane prediction bounding boxes relative to the input image.
6. The system of claim 1 , wherein the CNN is configured as a single shot detector or a Fast-RCNN.
7. The system of claim 1 , further comprising a camera mounted to the motor vehicle, the camera capturing the input image.
8. The system of claim 1 , wherein the lane line module fits a second order polynomial through center points of the multiple final lane bounding box predictions.
9. The system of claim 1 , wherein the CNN is trained using multiple annotated input images having lane bounding boxes centered on left lane markings and lane bounding boxes centered on right lane markings.
10. The system of claim 9 , wherein each of the multiple lane bounding boxes are centered on a separate portion of the left lane markings and the right lane markings.
11. A system for predicting object detection and lane detection for a motor vehicle, the system comprising:
a camera mounted to the motor vehicle, the camera capturing an input image having an object, a left lane marking, and a right lane marking;
a convolution neural network (CNN) receiving the input image, the CNN having:
a set of convolution and pooling layers (CPL's) trained to detect the object, the left lane marking, and the right lane marking from the input image, wherein the object is categorized into an object class, the left lane marking is categorized into a left lane marking class, and the right lane marking is categorized into a right lane marking class, the CPL's generating a number of feature maps;
a fully connected layer that receives the feature maps, the fully connected layer generating multiple object bounding box predictions for each of the object classes and multiple lane bounding box predictions for the left lane marking class and the right lane marking class from the feature maps;
a non-maximum suppression layer receiving the multiple object bounding box predictions and the multiple lane bounding box predictions and generating a final object bounding box prediction for the object class and generating multiple final lane bounding box predictions for each of the lane marking classes; and
a lane line module configured to generate lane lines from the multiple final lane bounding box predictions.
12. The system of claim 11 , wherein the multiple final lane bounding box predictions are centered on the lane markings.
13. The system of claim 12 , wherein the multiple final lane bounding box predictions each include a confidence value.
14. The system of claim 12 , wherein the multiple final lane bounding box predictions each include localization information that locate the multiple final lane prediction bounding boxes relative to the input image.
15. The system of claim 11 , wherein the CNN is configured as a single shot detector or a Fast-RCNN.
16. The system of claim 11 , wherein the lane line module fits a second order polynomial through center points of the multiple final lane bounding box predictions.
17. The system of claim 11 , wherein the CNN is trained using multiple annotated input images having lane bounding boxes centered on left lane markings and lane bounding boxes centered on right lane markings.
18. The system of claim 17 , wherein each of the multiple lane bounding boxes are centered on a separate portion of the left lane markings and the right lane markings.