Harvester systems and methods for automated and semi-automated filling of bins of receiving vehicles
Described herein are technologies that use LIDAR and computer vision to detect a location of a receiving vehicle relative to a forage harvester, fill levels of crop material within the receiving vehicle, and path and landing position of material expelled from the forage harvester and received by a bin of the receiving vehicle. The technologies use such information as feedback for operating the harvester or the receiving vehicle. Some embodiments detect ground level in front of the harvester or the receiving vehicle, and such information is used as feedback too. Some embodiments include a link to communicate the feedback to a GUI for user visualization of the feedback and semi-automated operations of the harvester or the receiving vehicle. For example, readings from LIDAR and a camera of the harvester detect a topography of the material deposited in the bin of the receiving vehicle, and a GUI outputs the topography.
1. An apparatus, comprising:
a camera, configured to capture image data of a receiving vehicle and a self-propelling vehicle moving the receiving vehicle that is near a harvester;
a LIDAR system, configured to:
scan in coordinates of parts of the receiving vehicle and the self-propelled vehicle; and
scan in coordinates of a distribution of crop material in the receiving vehicle; and
a computing system, configured to:
convert the scanned in coordinates of the receiving vehicle, the self-propelled vehicle and the distribution of crop material into a point cloud;
detect an image of the receiving vehicle from the image data;
generate a bounding box that surrounds the detected imaged of the receiving vehicle, wherein the bounding box is either a rectangular cuboid or a rectangle;
detect boundaries of the receiving vehicle using edge detection, within the generated bounding box;
estimate fill level based on the detected boundaries and the point cloud;
time-match the captured image data and the point cloud to provide time-matched data;
extract a receiving vehicle portion of the point cloud based on the time-matched data and the detected image of the receiving vehicle;
estimate dimensions and orientation of the receiving vehicle based on the point cloud; and
generate graphical representations of the estimated fill level and the estimated dimensions and orientation of the receiving vehicle to be displayed on a screen simultaneously.
2. The apparatus of claim 1 ,
wherein the LIDAR system is configured to:
scan in a crop material flow expelled from a spout of a discharge chute of the harvester;
wherein the computing system is configured to:
determine a crop material path by extrapolating points along an arc formed by the scanned in crop material flow;
determine a landing point of the expelled crop material flow by curve fitting the determined crop material path to a point of intersection with a surface of the receiving vehicle, based on the estimated dimensions and orientation of the receiving vehicle; and
generate a graphical representation of the determined crop material path and the determined landing point along with the estimated fill level and the estimated dimensions and orientation of the receiving vehicle to be displayed on the screen simultaneously.
3. The apparatus of claim 2 , wherein the computing system is configured to:
detect an image of the spout from the captured image data;
generate a second bounding box that surrounds the detected imaged of the spout;
detect boundaries of the spout using edge detection, within the generated second bounding box;
extract a spout portion of detected coordinates in the detected crop material flow, based on the time-matched data and the detected image of the spout;
estimate dimensions and orientation of the spout based on the extracted spout portion of the detected coordinates in the scanned in crop material flow; and
generate a graphical representation of the estimated dimensions and orientation of the spout along with the estimated fill level and the estimated dimensions and orientations of the receiving vehicle to be displayed on the screen simultaneously.
4. The apparatus of claim 1 , wherein the computing system is configured to:
detect an image of the self-propelled vehicle moving the receiving vehicle from the captured image data;
generate a second bounding box that surrounds the detected imaged of the self-propelled vehicle;
detect boundaries of the self-propelled vehicle using edge detection, within the generated second bounding box;
extract a moving vehicle portion of the point cloud, based on the time-matched data and the detected image of the self-propelled vehicle;
estimate dimensions and orientation of the self-propelled vehicle based on the extracted moving vehicle portion of the point cloud; and
generate a graphical representation of the estimated dimensions and orientation of the self-propelled vehicle along with the estimated fill level and the estimated dimensions and orientation of the receiving vehicle to be displayed on the screen simultaneously.