Systems and methods for determining trailer pose and coupler position for automated hitching
A method for determining a location and pose of at least one trailer for hitching to a vehicle includes: receiving at least one image of at least one trailer generated by a camera of the vehicle; locating the at least one trailer in the at least one image; determining a plurality of keypoints associated with at least one trailer in the image using at least one machine learning model; determining a location of a coupler of the at least one trailer relative to the vehicle and a pose of the at least one trailer relative to the vehicle based on the computed keypoints; and determining a path for maneuvering the vehicle into position for hitching the at least one trailer to the vehicle based on the location of the coupler of the at least one trailer and the pose of the at least one trailer.
1. A method for determining a location and pose of at least one trailer for hitching to a vehicle comprising:
receiving, by at least one processor, at least one image of at least one trailer generated from at least one camera of the vehicle;
locating, by the at least one processor, the at least one trailer in the at least one image;
determining, by the at least one processor, a plurality of keypoints associated with the at least one trailer in the image using at least one machine learning model;
determining, by the at least one processor, a location of a coupler of the at least one trailer relative to the vehicle and a pose of the at least one trailer relative to the vehicle based on the determined keypoints; and
determining, by the at least one processor, a path for maneuvering the vehicle into position for hitching the at least one trailer to the vehicle based on the location of the coupler of the at least one trailer and the pose of the at least one trailer,
wherein the keypoints comprise a first set of keypoints and a second set of keypoints, and determining the pose of the at least one trailer comprises:
generating, by the at least one processor, a first estimate of the pose of the at least one trailer based on the first set of keypoints;
generating, by the at least one processor, a second estimate of the pose of the at least one trailer based on the second set of keypoints; and
determining, by the at least one processor, the pose of the at least one trailer based on the first and second estimates,
wherein the first and second estimates of the pose of the at least one trailer are weighted based on a distance from the vehicle to the at least one trailer, and
wherein the location of the coupler and the pose of the at least one trailer are determined based on image data only captured by the at least one camera of the vehicle.
2. The method of claim 1 , wherein the first set of keypoints is associated with a tow bar of the at least one trailer and the second set of keypoints is associated with a body of the at least one trailer from which the tow bar extends.
3. The method of claim 2 , wherein the first estimate of the pose is weighted lower than the second estimate of the pose for a first distance between the vehicle and the to the at least one trailer and the first estimate of the pose is weighted higher than the second estimate of the pose for a second distance, less than the first distance, between the vehicle and the at least one trailer.
4. The method of claim 1 , further comprising updating, by the at least one processor, the location of the coupler of the at least one trailer and a pose of the at least one trailer based on new images generated as the vehicle maneuvers toward the at least one trailer.
5. The method of claim 1 , wherein the at least one image captures a plurality of trailers and keypoints are identified for the plurality of trailers.
6. The method of claim 5 , wherein the location of the coupler of the at least one trailer and the pose of the at least one trailer are determined for a trailer selected by a driver of the vehicle from among the plurality of trailers in the at least one image.
7. The method of claim 1 , wherein a single machine learning model locates the at least one trailer and determines the plurality of keypoints.
8. The method of claim 7 , wherein the single machine learning model comprises a keypoint detector.
9. The method of claim 1 , wherein the at least one image includes representations of a plurality of trailers, locating the at least one trailer comprises simultaneously locating each trailer in the at least one image, and determining the keypoints comprises determining keypoints for each trailer in the at least one image simultaneously.
10. The method of claim 1 , wherein the first and second estimates of the pose of the at least one trailer are weighted based on at least one of a brightness level of the at least one image, trailer type of the at least one trailer, and a size of the at least one trailer.
11. A system for determining a location and pose of at least one trailer for hitching to a vehicle, the system comprising one or more processors, memory, and one or more programs stored in the memory and including instructions for execution by the one or more processors for:
receiving at least one image of at least one trailer generated by a camera of the vehicle;
locating the at least one trailer in the at least one image;
determining a plurality of keypoints associated with at least one trailer in the image using at least one machine learning model;
determining a location of a coupler of the at least one trailer relative to the vehicle and a pose of the at least one trailer relative to the vehicle based on the determined keypoints; and
determining a path for maneuvering the vehicle into position for hitching the at least one trailer to the vehicle based on the location of the coupler of the at least one trailer and the pose of the at least one trailer,
wherein the keypoints comprise a first set of keypoints associated with a tow bar of the at least one trailer and a second set of keypoints associated with a body of the at least one trailer from which the tow bar extends,
wherein determining the pose of the at least one trailer comprises:
generating a first estimate of the pose of the at least one trailer based on the first set of keypoints;
generating a second estimate of the pose of the at least one trailer based on the second set of keypoints; and
determining the pose of the at least one trailer based on the first and second estimates,
wherein the location of the coupler and the pose of the at least one trailer are determined based on image data only captured by the at least one camera of the vehicle.
12. The system of claim 11 , wherein the first and second estimates of the pose of the at least one trailer are weighted based on a distance from the vehicle to the at least one trailer.
13. The system of claim 12 , wherein the first estimate of the pose is weighted lower than the second estimate of the pose for a first distance between the vehicle and the at least one trailer, and the first estimate of the pose is weighted higher than the second estimate of the pose for second distance, less than the first distance, between the vehicle and the at least one trailer.
14. The system of claim 11 , wherein the one or more programs including instructions for: updating the location of the coupler of the at least one trailer and a pose of the at least one trailer based on new images generated as the vehicle maneuvers toward the at least one trailer.
15. The system of claim 11 , wherein the at least one image captures a plurality of trailers and keypoints are identified for the plurality of trailers.
16. The system of claim 15 , wherein the location of the coupler of the at least one trailer and the pose of the at least one trailer are determined for a trailer selected by a driver of the vehicle from among the plurality of trailers in the at least one image.
17. The system of claim 11 , wherein a single machine learning model locates the at least one trailer and determines the plurality of keypoints.
18. The system of claim 17 , wherein the single machine learning model comprises a keypoint detector.
19. The system of claim 11 , wherein the at least one image includes representations of a plurality of trailers, and wherein the at least one processor comprises a trained machine learning model which locates each trailer representation in the at least one image and determines the keypoints.
20. The system of claim 11 , wherein the at least one image includes representations of a plurality of trailers, and the plurality of trailers are located in the at least one image simultaneously, and the keypoints for each trailer in the at least one image are determined simultaneously.
21. The system of claim 11 , wherein the at least one image includes representations of a plurality of trailers, and wherein the at least one processor comprises a first trained neural network or machine learning model which defines a bounding box surrounding each trailer in the at least one image, and further comprises a second trained neural network or machine learning model which determines the keypoints of the at least one trailer within each bounding box.
22. The system of claim 11 , wherein the at least one image includes representations of a plurality of trailers, the at least one processor comprises one trained neural network or one machine learning model which defines a bounding box surrounding each trailer in the at least one image and determines the keypoints of the at least one trailer within each bounding box.