Trailer angle estimation using machine learning
In various examples, a trailer angle may be estimated using one or more machine learning models to predict one or more keypoints on the center axis of the trailer drawbar (e.g., a keypoint representing the drawbar junction around which the drawbar pivots, one or more other keypoints along the center axis), back-projecting the predicted keypoint(s) onto a three-dimensional (3D) representation of the ground, and calculating the angle between the longitudinal axis of the towing vehicle and a line or ray formed by or fitted to the projected keypoints. The trailer angle may be estimated at any frame rate. For each frame, keypoints may be predicted from that frame and/or optical flow or some other type of feature tracking may be used to propagate predicted keypoint(s) from a preceding frame in lieu of predicting keypoint(s), and the resulting keypoint(s) may be used to estimate the trailer angle for that frame.
1 . A processor comprising:
one or more processing units to:
generate, based at least on one or more machine learning models processing a representation of sensor data corresponding to a view of a drawbar connecting a trailer to an ego-machine, a representation of three or more points substantially along a center axis of the drawbar;
generate, based at least on an estimated ground projection of the center axis of the drawbar corresponding to the three or more points, an estimated trailer angle between a first axis of the ego-machine and a second axis of the trailer; and
control one or more operations of the ego-machine based at least on the estimated trailer angle.
2 . The processor of claim 1 , wherein the one or more processing units are further to generate the estimated trailer angle based at least on back-projecting the three or more points into a three-dimensional (3D) coordinate system to identify 3D locations corresponding to the estimated ground projection of the center axis of the drawbar and calculating the estimated trailer angle as an angle between a longitudinal axis of the ego-machine and the estimated ground projection of the center axis of the drawbar.
3 . The processor of claim 1 , wherein the three or more points substantially along the center axis of the drawbar comprise a first keypoint representing a junction around which the drawbar pivots and one or more other keypoints.
4 . The processor of claim 1 , wherein the three or more points substantially along the center axis of the drawbar comprise a first fixed keypoint representing a junction around which the drawbar pivots and one or more predicted keypoints.
5 . The processor of claim 1 , wherein the one or more processing units are further to generate the estimated trailer angle based at least on a three-dimensional (3D) representation of the estimated ground projection of the center axis of the drawbar corresponding to a line or ray fitted to a ground projection of the three or more points.
6 . The processor of claim 1 , wherein the sensor data represents a two-dimensional (2D) view of the drawbar, and the three or more points along the center axis of the drawbar comprise two or more 2D points detected in the 2D view by the one or more machine learning models.
7 . The processor of claim 1 , wherein the one or more machine learning models comprise a neural network with an output channel for each individual point of the three or more points substantially along the center axis of the drawbar, and the output channel for each point is configured to predict classification data representing a likelihood that each individual pixel, of one or more pixels of the representation of the sensor data, depicts the individual point associated with the center axis of the drawbar.
8 . The processor of claim 1 , wherein the sensor data comprises image data generated using a rear-facing camera of the ego-machine.
9 . The processor of claim 1 , wherein the one or more processing units are further to repetitively generate the estimated trailer angle at a designated frame rate based at least on updated detections of the three or more points substantially along the center axis of the drawbar represented in corresponding frames of the sensor data.
10 . The processor of claim 1 , wherein the one or more processing units are further to propagate at least one detected point of the three or more points substantially along the center axis of the drawbar to a corresponding location in a subsequent frame using optical flow.
11 . The processor of claim 1 , wherein the operations of the ego-machine that are based at least on the estimated trailer angle comprise one or more of path planning, obstacle avoidance, presenting a recommended path on a display visible to an operator of the ego-machine, or self-parking.
12 . The processor of claim 1 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for the autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
13 . A system comprising one or more processing units to control one or more operations of an ego-machine using a trailer kink angle estimated based at least on an estimated ground projection of a center axis of a drawbar connecting a trailer to the ego-machine, the estimated ground projection generated using three or more points detected substantially along the center axis of the drawbar using one or more machine learning models.
14 . The system of claim 13 , wherein the one or more processing units are further to estimate the trailer kink angle based at least on back-projecting the three or more points into a three-dimensional (3D) coordinate system to identify 3D locations corresponding to the estimated ground projection of the center axis of the drawbar and calculating the estimated trailer kink angle as an angle between a longitudinal axis of the ego-machine and the estimated ground projection of the center axis of the drawbar.
15 . The system of claim 13 , wherein the three or more points substantially along the center axis of the drawbar comprise a first keypoint representing a junction around which the drawbar pivots and one or more other keypoints.
16 . The system of claim 13 , wherein the using the one or more machine learning models to detect the three or more points comprises applying sensor data representing a two-dimensional (2D) view of the drawbar to the one or more machine learning models to detect the three or more points in the 2D view.
17 . The system of claim 13 , wherein the operations of the ego-machine controlled using the trailer kink angle comprise one or more of path planning, obstacle avoidance, presenting a recommended path on a display visible to an operator of the ego-machine, or self-parking.
18 . The system of claim 13 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for the autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
19 . A method comprising:
generating, using one or more machine learning models and based at least on a two-dimensional (2D) view of a drawbar connecting a trailer to an ego-machine, a representation of three or more 2D points in the 2D view along the drawbar;
generating, based at least on an estimated ground projection of a center axis of the drawbar corresponding to the three or more 2D points, an estimated trailer angle between a first axis of the ego-machine and a second axis of the trailer; and
controlling one or more operations of the ego-machine based at least on the estimated trailer angle.
20 . The method of claim 19 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for the autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.