Long-term visual trailer tracker for vehicle-trailer angle estimation
A method for visual trailer tracking of vehicle-trailer angle estimation, including: (a) camera receiving images, each most recently received image being identified as a current image, replacing a previously identified current image; (b) during a driving maneuver, extracting visual features from the current image, calculating a trailer angle from the current image, and storing the current image as a reference image, the trailer angle as a trailer angle reference and the extracted visual features as visual reference features; and (c) matching the visual features from the current image with the visual reference features. When a number of matches is below a threshold, repeating operation (c) to update the reference image, the trailer angle reference and the visual reference features. The method includes calculating a trailer angle between a trailer representation in the current image and a trailer representation in the reference image based in part upon the trailer angle reference.
1. A method for visual trailer tracking for vehicle-trailer angle estimation, the method comprising:
(a) receiving, by data processing hardware, images from a camera positioned on a rear portion of the tow vehicle facing a trailer connected to the tow vehicle, each most recently received image being identified as a current image, replacing a previously identified current image;
(b) during a driving maneuver extracting, by the data processing hardware, visual features of a representation of the trailer from the current image, calculating, by the data processing hardware, a trailer angle of the trailer relative to the tow vehicle from the current image and storing in memory coupled to the data processing hardware the current image as a reference image, the trailer angle as a trailer angle reference and the extracted visual features as visual reference features;
(c) subsequent to (b) and after identifying, by the data processing hardware, a recently received image as the current image, extracting, by the data processing hardware, visual reference features of a representation of the trailer from the current image and matching, by the data processing hardware, the visual features from the current image with the visual reference features of the reference image;
(d) when the matching is below a first predetermined threshold, repeating (c) by the data processing hardware to update the reference image, the trailer angle reference and the visual reference features;
(e) calculating, by the data processing hardware, a trailer angle between the trailer representation in the current image and the trailer representation in the reference image based in part upon the trailer angle reference; and
(f) sending, from the data processing hardware to one or more vehicle systems, the calculated trailer angle.
2. The method of claim 1 , further comprising prior to (b), receiving or determining, by the data processing hardware, a bounding box, wherein extracting the visual features of the trailer representation in (b) comprises extracting the visual features within the bounding box of the current image, and the method further comprises updating, by the data processing hardware, the bounding box to contain the matched visual features.
3. The method of claim 1 , further comprising:
prior to (b), during a zero angle maneuver as the tow vehicle and trailer move in a largely straight path, determining, at the data processing hardware, visual features of the trailer representation from the images received during the zero angle maneuver and storing in the memory the visual features as zero angle reference features.
4. The method of claim 3 , further comprising extracting, by the data processing hardware, visual features of the trailer representation from the current image, comparing the extracted visual features of the current image with the zero angle reference features, and when a distance between the comparison is below a second predetermined threshold, setting the trailer angle reference to zero and the trailer angle to zero.
5. The method of claim 1 , wherein matching the visual features from the current image with the visual reference features comprises performing a correlation, a Brute-Force matcher or a FLANN based matcher on the visual features from the current image with the visual reference features.
6. The method of claim 1 , wherein calculating the trailer angle in operation (e) is based in part upon a depth of the visual features of the trailer representation in the current image and the visual reference features.
7. The system of claim 6 , wherein the operations further comprise prior to (b), receiving or determining a bounding box, wherein extracting the visual features of the trailer representation in (b) comprising extracting the visual features within the bounding box of the current image, and the method further comprises updating the bounding box to contain the matched visual features.
8. The system of claim 6 , wherein the operations further comprise:
prior to (b), during a zero angle maneuver as the tow vehicle and trailer move in a largely straight path, determining, at the data processing hardware, visual features of the trailer representation from the images received during the zero angle maneuver and storing in the memory hardware the visual features as zero angle reference features.
9. The system of claim 8 , wherein the operations further comprise extracting visual features of the trailer representation from the current image, comparing the extracted visual features of the current image with the zero angle reference features, and when a distance between the comparison is below a second predetermined threshold, setting the trailer angle reference to zero and the trailer angle to zero.
10. A system for visual trailer tracking of vehicle-trailer angle estimation, the system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
(a) receiving images from a camera positioned on a rear portion of the tow vehicle facing a trailer connected to the tow vehicle, each most recently received image being identified as a current image, replacing a previously identified current image;
(b) during a driving maneuver extracting visual features of a representation of the trailer from the current image, calculating a trailer angle of the trailer relative to the tow vehicle from the current image and storing in the memory hardware coupled to the data processing hardware the current image as a reference image, the trailer angle as a trailer angle reference and the extracted visual features as visual reference features;
(c) subsequent to (b) and after identifying a recently received image as the current image, extracting visual reference features of a representation of the trailer from the current image and matching the visual features from the current image with the visual reference features of the reference image;
(d) when the matching is below a first predetermined threshold, repeating, (c) by the data processing hardware to update the reference image, the trailer angle reference and the visual reference features;
(e) calculating a trailer angle between the trailer representation in the current image and the trailer representation in the reference image based in part upon the trailer angle reference; and
(f) sending, from the data processing hardware to one or more vehicle systems, the calculated trailer angle.
11. The system of claim 10 , wherein matching the visual features from the current image with the visual reference features comprises performing a correlation, a Brute-Force matcher, or a FLANN based matcher on the visual features from the current image with the visual reference features.
12. The system of claim 10 , wherein calculating the trailer angle in operation (e) is based in part upon a depth of the visual features of the trailer representation in the current image and the visual reference features.
13. The system of claim 10 , further comprising the camera positioned on the rear portion of the tow vehicle, the camera being communicatively coupled to the data processing hardware.