Systems and methods for training a camera-based perception model using machine learning
Systems and methods include detecting obstacles and drivable areas by an autonomous vehicle by inputting image and map data into a neural network to extract feature vectors. A transformer encoder converts these vectors from camera space to Bird's Eye View (BEV) space. A detection head identifies objects, and a segmentation head generates a BEV map showing objects and drivable surfaces. Attributes from both heads are compared, and the segmentation head's weights are updated accordingly, resulting in an updated BEV segmentation map output by the updated segmentation head.
1 . A method for detecting obstacles and drivable area by an autonomous vehicle in motion, the method comprising:
inputting both image data collected from at least one camera and map data into neural network feature extractors configured to output a first set of feature vectors;
encoding, using a transformer encoder, the first set of feature vectors into a second set of Bird's Eye View (BEV) feature vectors, wherein the transformer encoder converts information in the first set of feature vectors from a camera space to a BEV space based on spatial relationships relative to the autonomous vehicle;
identifying, using a detection head, at least one object in the second set of BEV feature vectors;
generating, using a segmentation head, a BEV segmentation map from the second set of BEV feature vectors, wherein the BEV segmentation map identifies the at least one object and drivable surfaces around the autonomous vehicle;
comparing attributes of the at least one object detected using the detection head and attributes of the at least one object detected using the segmentation head;
updating weights of the segmentation head based on the comparison; and
outputting, via the updated segmentation head, an updated BEV segmentation map.
2 . The method of claim 1 , further comprising generating a navigation recommendation using a path-planning model configured to process the updated BEV segmentation map.
3 . The method of claim 1 , wherein the at least one camera is operably coupled with the autonomous vehicle in motion.
4 . The method of claim 2 , wherein the path-planning model is configured to calculate a plurality of possible paths using a cost model, and wherein generating the navigation recommendation comprises selecting a lowest cost path from among the plurality of possible paths based on a change in cost assigned by the path-planning model.
5 . The method of claim 4 , wherein selecting the lowest cost path includes applying an optimizer using a Monte Carlo approximation.
6 . The method of claim 2 , wherein the path-planning model is a Model Predictive Path Integral (MPPI) module.
7 . The method of claim 1 , wherein the transformer encoder is a spatial cross-attention transformer.
8 . The method of claim 1 , further comprising:
during a training stage of the segmentation head, generating a plurality of training batches, wherein each of the plurality of training batches includes at least one sample with objects to detect.
9 . The method of claim 1 , wherein attributes of the at least one object include visual features of the at least one object and a position of the at least one object.
10 . The method of claim 1 , wherein comparing the attributes of the at least one object detected using the detection head and the attributes of the at least one object detected using the segmentation head comprises calculating a loss function between both attributes.
11 . The method of claim 1 , wherein the at least one object is one or more of: a driving obstacle, a landmark, a pedestrian, a sign, a traffic light, another vehicle.
12 . A system for detecting obstacles and drivable area by an autonomous vehicle in motion, comprising:
at least one memory;
at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
input both image data collected from at least one camera and map data into neural network feature extractors configured to output a first set of feature vectors;
encode, using a transformer encoder, the first set of feature vectors into a second set of Bird's Eye View (BEV) feature vectors, wherein the transformer encoder converts information in the first set of feature vectors from a camera space to a BEV space based on spatial relationships relative to the autonomous vehicle;
identify, using a detection head, at least one object in the second set of BEV feature vectors;
generate, using a segmentation head, a BEV segmentation map from the second set of BEV feature vectors, wherein the BEV segmentation map identifies the at least one object and drivable surfaces around the autonomous vehicle;
compare attributes of the at least one object detected using the detection head and attributes of the at least one object detected using the segmentation head;
update weights of the segmentation head based on the comparison; and
output, via the updated segmentation head, an updated BEV segmentation map.
13 . The system of claim 12 , wherein the at least one hardware processor is further configured to generate a navigation recommendation using a path-planning model configured to process the updated BEV segmentation map.
14 . The system of claim 12 , wherein the at least one camera is operably coupled with the autonomous vehicle in motion.
15 . The system of claim 13 , wherein the path-planning model is configured to calculate a plurality of possible paths using a cost model, and wherein generating the navigation recommendation comprises selecting a lowest cost path from among the plurality of possible paths based on a change in cost assigned by the path-planning model.
16 . The system of claim 15 , wherein the at least one hardware processor is further configured to select the lowest cost path by applying an optimizer using a Monte Carlo approximation.
17 . The system of claim 13 , wherein the path-planning model is a Model Predictive Path Integral (MPPI) module.
18 . The system of claim 12 , wherein the transformer encoder is a spatial cross-attention transformer.
19 . The system of claim 12 , wherein the at least one hardware processor is further configured to:
during a training stage of the segmentation head, generate a plurality of training batches, wherein each of the plurality of training batches includes at least one sample with objects to detect.
20 . A non-transitory computer readable medium storing thereon computer executable instructions for detecting obstacles and drivable area by an autonomous vehicle in motion, including instructions for:
inputting both image data collected from at least one camera and map data into neural network feature extractors configured to output a first set of feature vectors;
encoding, using a transformer encoder, the first set of feature vectors into a second set of Bird's Eye View (BEV) feature vectors, wherein the transformer encoder converts information in the first set of feature vectors from a camera space to a BEV space based on spatial relationships relative to the autonomous vehicle;
identifying, using a detection head, at least one object in the second set of BEV feature vectors;
generating, using a segmentation head, a BEV segmentation map from the second set of BEV feature vectors, wherein the BEV segmentation map identifies the at least one object and drivable surfaces around the autonomous vehicle;
comparing attributes of the at least one object detected using the detection head and attributes of the at least one object detected using the segmentation head;
updating weights of the segmentation head based on the comparison; and
outputting, via the updated segmentation head, an updated BEV segmentation map.