Path detection for autonomous machines using deep neural networks
In various examples, a deep learning solution for path detection is implemented to generate a more abstract definition of a drivable path without reliance on explicit lane-markings—by using a detection-based approach. Using approaches of the present disclosure, the identification of drivable paths may be possible in environments where conventional approaches are unreliable, or fail—such as where lane markings do not exist or are occluded. The deep learning solution may generate outputs that represent geometries for one or more drivable paths in an environment and confidence values corresponding to path types or classes that the geometries correspond. These outputs may be directly useable by an autonomous vehicle—such as an autonomous driving software stack—with minimal post-processing.
1. A method comprising:
receiving image data representative of a field of view of an image sensor;
applying an image represented by the image data to a neural network;
computing, using the neural network and for an individual anchor point of an anchor line associated with the image, a set of delta values between the individual anchor point and a vertex of one or more vertices of a predicted path in the field of view, an individual delta value of the set of delta values corresponding to a distance from the individual anchor point to the vertex of the one or more vertices;
generating a geometry for the predicted path with respect to the image based at least in part on the set of delta values;
determining, based at least in part on a confidence value computed using the neural network, a path class that corresponds to the predicted path; and
using the geometry and the path class to perform one or more operations by an autonomous vehicle.
2. The method of claim 1 , wherein the individual anchor point is a first individual anchor point, the anchor line is a first anchor line, the set of delta values are a first set of delta values, the geometry is a first geometry, the predicted path is a first predicted path, the vertex is a first vertex, the one or more vertices are one or more first vertices, the confidence value is a first confidence value, and the method further comprises:
computing, using the neural network and for a second individual anchor point of a second anchor line associated with the image, a second set of delta values between the second individual anchor point and a second vertex of one or more second vertices of a second predicted path, an individual second delta value of the second set of delta values corresponding to a distance from the second individual anchor point to the second vertex of the one or more second vertices;
generating a second geometry for the second predicted path with respect to the image based at least in part on the second set of delta values; and
determining, based at least in part on a second confidence value computed using the neural network being less than the first confidence value, that the second predicted path does not correspond to the path class.
3. The method of claim 1 , wherein the neural network computes one or more sets of delta values for an individual anchor line of a plurality of anchor lines, and further computes one or more confidence values representative of a likelihood that the individual anchor line of the plurality of anchor lines corresponds to an individual path class of a plurality of path classes.
4. The method of claim 1 , wherein the anchor line is one of a linear anchor line or a cubic polynomial anchor line.
5. A method comprising:
applying image data to a neural network, the image data representing an image;
computing, using the neural network and for an individual anchor point of one or more anchor points associated with the image data:
one or more delta values representative of one or more distances between a location within the image that is associated with the individual anchor point and one or more locations within the image that are associated with one or more vertices of a predicted path; and
one or more confidence values representative of a confidence that the predicted path corresponds to an individual path class of one or more path classes;
generating a geometry for the predicted path based at least in part on the one or more delta values;
assigning a path class of the one or more path classes to the predicted path based at least in part on the one or more confidence values; and
using the geometry and the path class to perform one or more operations by a vehicle within a physical environment.
6. The method of claim 5 , wherein the one or more anchor points include a plurality of anchor points arranged in a grid pattern at static pixel locations with respect to the image represented by the image data.
7. The method of claim 5 , wherein the one or more delta values include a set of delta values, the set of delta values including one or more first delta values between the individual anchor point and the one or more vertices in a first direction and one or more second delta values between the individual anchor point and the one or more vertices in a second direction.
8. The method of claim 5 , wherein the one or more path classes include a plurality of path classes, the plurality of path classes including a vehicle path of the vehicle and one or more related paths relative to the vehicle path.
9. The method of claim 5 , wherein the assigning the path class to the predicted path includes determining that a confidence value of the one or more confidence values for the path class for the individual anchor point is greater than one or more second confidence values for the path class for one or more second anchor points.
10. The method of claim 5 , wherein the one or more vertices correspond to one or more points of at least one polyline, and the geometry represents a polygon formed by connecting the one or more points to form the at least one polyline.
11. The method of claim 5 , wherein the predicted path includes a first polyline and a second polyline laterally spaced from the first polyline, and the method further comprises:
computing a center polyline extending between and along the first polyline and the second polyline,
wherein the using the geometry includes using the center polyline.
12. The method of claim 5 , wherein the predicted path includes a center polyline, and the method further comprises:
computing a first polyline and a second polyline based at least in part on the center polyline, the first polyline and the second polyline being spaced substantially equidistant from the one or more vertices of the center polyline,
wherein the using the geometry includes using the first polyline and the second polyline as input to perform the one or more operations by the vehicle.
13. A processor comprising:
processing circuitry to:
based at least in part on sensor data generated using one or more sensors of a machine, compute, using a neural network and for an individual anchor point of a plurality of one or more anchor points associated with the sensor data:
a set of delta values representative of distances between the individual anchor point and a vertex of one or more vertices of a predicted path; and
one or more confidence values representative of a confidence that the predicted path corresponds to one or more path classes;
generate a geometry for the predicted path based at least in part on the set of delta values;
assign a path class of the one or more path classes to the predicted path based at least in part on the one or more confidence values; and
use the geometry and the path class to perform one or more operations by the machine within a physical environment.
14. The processor of claim 13 , wherein the one or more anchor points include a plurality of anchor points arranged in a grid pattern at static pixel locations with respect to an image represented by the sensor data.
15. The processor of claim 13 , wherein the set of delta values includes one or more first delta values between the individual anchor point and the one or more vertices in a first direction and one or more second delta values between the individual anchor point and the one or more vertices in a second direction.
16. The processor of claim 13 , wherein the one or more path classes include a plurality of path classes, the plurality of path classes including a vehicle path of the vehicle and one or more related paths relative to the vehicle path.
17. The processor of claim 13 , wherein the assigning the path class to the predicted path includes determining that a confidence value of the one or more confidence values for the path class for the individual anchor point is greater than one or more additional confidence values for the path class for one or more additional anchor points.
18. The processor of claim 13 , wherein:
the one or more anchor points include a plurality of anchor points;
the one or more path classes include a plurality of path classes;
the neural network computes a respective path class for each of the plurality of anchor points; and
each respective path class of the plurality of path classes is assigned to a respective anchor point having a highest confidence value for the respective path class.
19. The processor of claim 13 , wherein the one or more vertices correspond to one or more points of at least one polyline, and the geometry represents a polygon formed by connecting the one or more points to form the at least one polyline.
20. The method of claim 5 , wherein:
the location within the image that is associated with the individual anchor point is a pixel location within the image that is associated with the individual anchor point;
the one or more locations within the image that are associated with the one or more vertices of the predicted path are one or more pixel locations within the image that are associated with the one or more vertices of the predicted path; and
the one or more distances are one or more pixel distances between the pixel location within the image that is associated with the individual anchor point and the one or more pixel locations within the image that are associated with the one or more vertices of the predicted path.