Real-time trailer coupler localization and tracking

1. A method of detecting and localizing a trailer coupler of a trailer, the method comprising:

receiving, at data processing hardware, images from a camera positioned on a back portion of a tow vehicle and in communication with the data processing hardware;

determining, by the data processing hardware, a region of interest within the images, the region of interest including a representation of the trailer coupler;

determining, by the data processing hardware, a camera plane where the camera moves based on the received images;

determining, by the data processing hardware, a road plane based on the received images;

determining, by the data processing hardware, a three-dimensional (3D) point cloud representing objects inside the region of interest and within the camera plane and the road plane;

receiving, at the data processing hardware, sensor data from at least one of wheel encoder, acceleration and wheel angle sensors, and an inertial measurement unit in communication with the data processing hardware;

determining, at the data processing hardware, a coupler location of the trailer coupler based on the 3D point cloud and the sensor data, the coupler location being in real world coordinates; and

sending, from the data processing hardware to a drive system, instructions causing the tow vehicle to autonomously drive along a path in a rearward direction towards the coupler location.

2. The method of claim 1 , wherein determining the region of interest within the images includes:

sending, from the data processing hardware to a display, instructions to display the received images; and

receiving, at the data processing hardware, a user selection of the region of interest.

3. The method of claim 1 , further comprising:

projecting, by the data processing hardware, points associated with the 3D point cloud onto the camera plane or the road plane;

determining, by the data processing hardware, a distance between each point and the camera:

when the points associated with the 3D point cloud are projected onto the camera plane, determining the distance between each point and a center of the camera; and

when the points associated with the 3D point cloud are projected onto the road plane, determining the distance between each point and a projection of the center of the camera onto the road plane; and

determining, by the data processing hardware, a shortest distance based on the determined distances, a projection of the 3D point associated with the shortest distance onto the received images is representative of a coupler pixel location within the images, wherein the coupler location is based on the coupler pixel location.

4. The method of claim 3 , further comprising:

determining, by the data processing hardware, a coupler height based on a distance between the 3D point associated with the shortest distance and the road plane, wherein the coupler location includes the coupler height.

5. The method of claim 1 , further comprising:

determining, by the data processing hardware, a first distance between the trailer coupler and the camera based on the 3D point cloud; and

determining, by the data processing hardware, a second distance between the trailer coupler and a vehicle tow ball based on the first distance less than a longitudinal distance between the camera and a vehicle hitch ball;

wherein the path is based on the second distance.

6. The method of claim 1 , wherein determining the point cloud of the region of interest comprises executing one of a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, and a Structure from Motion (SfM) algorithm.

7. The method of claim 1 , wherein determining the camera plane comprises:

determining, by the data processing hardware, at least three three-dimensional positions of the rear camera from the received images; and

determining, by the data processing hardware, the camera plane based on the at least three three-dimensional positions.

8. The method of claim 1 , wherein determining the road plane comprises:

determining a height of the camera from a road supporting the tow vehicle; and

shifting the camera plane by the height of the camera.

9. The method of claim 1 , wherein determining the road plane comprises:

extracting, by the data processing hardware, at least three feature points from the images including a road;

associating, by the data processing hardware, a point of the 3D point cloud with each of the feature points; and

determining, by the data processing hardware, the road plane based on the at least three points of the 3D point cloud associated with the at least three feature points.

10. The method of claim 9 , wherein determining the camera plane comprises:

determining, by the data processing hardware, a height of the camera from the road; and

shifting, by the data processing hardware, the road plane by the height of the camera.

11. A system for detecting and localizing a trailer coupler of a trailer, the system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving one or more images from a camera positioned on a back portion of a tow vehicle and in communication with the data processing hardware;

determining a region of interest within the images, the region of interest including a representation of the trailer coupler;

determining a camera plane where the camera moves based on the received images;

determining a road plane based on the received images;

determining a three-dimensional (3D) point cloud representing objects inside the region of interest and within the camera plane and the road plane;

receiving sensor data from at least one of wheel encoder, acceleration and wheel angle sensors, and an inertial measurement unit in communication with the data processing hardware;

determining a coupler location of the trailer coupler based on the 3D point cloud and the sensor data, the coupler location being in real world coordinates; and

sending instructions to a drive system causing the tow vehicle to autonomously drive along a path in a rearward direction towards the coupler location.

12. The system of claim 11 , wherein determining the region of interest within the images includes:

sending, instructions to a display causing the display of the received images; and

receiving a user selection of the region of interest.

13. The system of claim 11 , wherein the operations further comprise:

projecting points associated with the 3D point cloud onto the camera plane or the road plane;

determining a distance between each point and the camera:

when the points associated with the 3D point cloud are projected onto the camera plane, determining the distance between each point and a center of the camera; and

when the points associated with the 3D point cloud are projected onto the road plane, determining the distance between each point and a projection of the center of the camera onto the road plane; and

determining a shortest distance based on the determined distances, a projection of the 3D point associated with the shortest distance onto the received images is representative of a coupler pixel location within the images, wherein the coupler location is based on the coupler pixel location.

14. The system of claim 13 , wherein the operations further comprise:

determining a coupler height based on a distance between the 3D point associated with the shortest distance and the road plane, wherein the coupler location includes the coupler height.

15. The system of claim 11 , wherein the operations further comprise:

determining a first distance between the trailer coupler and the camera based on the 3D point cloud; and

determining a second distance between the trailer coupler and a vehicle tow ball based on the first distance less than a longitudinal distance between the camera and a vehicle hitch ball;

wherein the path is based on the second distance.

16. The system of claim 11 , wherein determining the 3D point cloud of the region of interest comprises executing one of a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, and a Structure from Motion (SfM) algorithm.

17. The system of claim 11 , wherein determining the camera plane comprises:

determining, by the data processing hardware, at least three three-dimensional positions of the rear camera from the received images; and

determining, by the data processing hardware, the camera plane based on the at least three three-dimensional positions.

18. The system of claim 11 , wherein determining the road plane comprises:

determining a height of the camera from a road supporting the tow vehicle; and

shifting the camera plane by the height of the camera.

19. The system of claim 11 , wherein determining the road plane comprises:

extracting at least three feature points from the images including a road;

associating a point of the 3D point cloud with each of the feature points; and

determining the road plane based on the at least three points of the 3D point cloud associated with the at least three feature points.

20. The system of claim 19 , wherein determining the camera plane comprises:

determining a height of the camera from the road; and

shifting the road plane by the height of the camera.