System and method for autonomous surveillance of an agricultural machine
In one aspect, an agricultural surveillance platform useful to identify an operating condition of an agricultural machine may include an autonomous sensor frame structured to fly in formation with an agricultural machine and detect an operating condition of the agricultural machine. The agricultural surveillance platform may include a propulsion system to provide propulsive power to the autonomous sensor frame, and an inertial measurement unit configured to generate flight data measurements. The agricultural surveillance platform may also include an object detection sensor having a field of view sized to capture an image scene that includes the agricultural machine and generate scene data representing the image scene. The agricultural surveillance platform may also include a sensor frame controller configured to maintain a position of the autonomous sensor frame about the agricultural machine and generate autonomous sensor frame data that includes plugged data indicative of a plugged operating condition of the agricultural machine.
1 . An agricultural surveillance platform useful to identify an operating condition of an agricultural machine, the agricultural surveillance platform comprising:
an autonomous sensor frame structured to fly in formation with an agricultural machine and detect an operating condition of the agricultural machine, the autonomous sensor frame having:
a propulsion system structured to provide propulsive power to the autonomous sensor frame;
an inertial measurement unit configured to generate flight data measurements;
an object detection sensor affixed to the autonomous sensor frame and having a field of view sized to capture an image scene that includes a ground-engaging tool of the agricultural machine, the object detection sensor structured to generate scene data representing the image scene; and
a sensor frame controller configured to:
receive the flight data measurements;
receive the scene data;
generate a control command based at least in part on a relative orientation of the agricultural machine identified in the scene data and the flight data measurements, the control command being useful to modulate the propulsion system and maintain a position of the autonomous sensor frame about the ground-engaging tool of the agricultural machine; and
generate autonomous sensor frame data based at least in part on the scene data, the autonomous sensor frame data including plugged data indicative of a plugged operating condition of the ground-engaging tool of the agricultural machine.
2 . The agricultural surveillance platform of claim 1 , wherein the plugged data includes data associated with a likelihood of the plugged operating condition.
3 . The agricultural surveillance platform of claim 1 , wherein the sensor frame controller is configured to generate the plugged data by:
identifying an implement of the agricultural machine in the scene data having the ground-engaging tool;
comparing a current condition of the implement identified in the scene data to a baseline condition for the implement; and
generating the plugged data indicative of the plugged operating condition of the agricultural machine when the current condition of the implement identified in the scene data differs by a predetermined threshold from the baseline condition for the implement.
4 . The agricultural surveillance platform of claim 3 , wherein the plugged data includes an indication of at least one of a confidence of the plugged operating condition or a severity of the plugged operating condition based at least in part on a difference between the current condition of the implement and the baseline condition for the implement.
5 . The agricultural surveillance platform of claim 1 , wherein the sensor frame controller is further structured to receive agricultural machine data indicative of a global position of the agricultural machine,
wherein the sensor frame controller is structured to generate the control command further based at least in part on the global position of the agricultural machine.
6 . The agricultural surveillance platform of claim 1 , wherein the sensor frame controller is structured to generate a flight path command based on the relative orientation of the agricultural machine identified in the scene data, the flight path command being configured to guide the autonomous sensor frame from its current position to a different position relative to the agricultural machine.
7 . The agricultural surveillance platform of claim 6 , wherein the flight path command is further based on a guidance flight path, the guidance flight path defining a standoff distance and height to be maintained by the autonomous sensor frame relative to the agricultural machine.
8 . The agricultural surveillance platform of claim 6 , wherein the flight path command is further based on a guidance flight path, the guidance flight path defining a regularly repeating pattern for switching between monitoring a first side of the agricultural machine and a second side of the agricultural machine along a lateral direction.
9 . The agricultural surveillance platform of claim 1 , wherein the sensor frame controller is configured to generate the plugged data by:
identifying an area immediately behind at least a portion of the agricultural machine including the ground-engaging tool in the scene data;
comparing a current condition in the area identified in the scene data to a baseline condition; and
generating the plugged data indicative of the plugged operating condition of the agricultural machine when the current condition differs by a predetermined threshold from the baseline condition.
10 . The agricultural surveillance platform of claim 1 , wherein the sensor frame controller is further configured to receive an input from an operator associated with needing to inspect the ground-engaging tool,
wherein the sensor frame controller is configured to generate the control command further based at least in part on the input from the operator.
11 . An agricultural surveillance system comprising:
an agricultural machine configured to work soil, the agricultural machine comprising:
an agricultural machine controller configured to control a state of operation of the agricultural machine; and
an autonomous sensor frame configured to navigate around the agricultural machine, the autonomous sensor frame including:
a propulsion system for providing propulsive power to the autonomous sensor frame;
an object detection sensor having a field of view sized to capture an image scene that includes a ground-engaging tool of the agricultural machine, the object detection sensor structured to generate a scene data representing the image scene;
a sensor frame controller configured to:
receive the scene data;
generate a flight path command for controlling the propulsion system based at least in part on a relative orientation of the agricultural machine identified in the scene data; and
generate an autonomous sensor frame data that includes a plugged data indicative of a plugged operating condition of the ground-engaging tool of the agricultural machine based at least in part on the scene data,
wherein the agricultural machine controller is configured to alter the state of operation of the agricultural machine based on the plugged operating condition of the ground-engaging tool of the agricultural machine.
12 . The agricultural surveillance system of claim 11 , wherein the sensor frame controller generates the flight path command based further at least in part on position data from the agricultural machine to change a position of the autonomous sensor frame to rendezvous with the agricultural machine, the position data from the agricultural machine being indicative of a global position of the agricultural machine.
13 . The agricultural surveillance system of claim 11 , wherein the object detection sensor is affixed to a gimballed platform structured to move relative to the autonomous sensor frame.
14 . The agricultural surveillance system of claim 11 , wherein the sensor frame controller includes an object avoidance controller structured to generate an object avoidance control command based on the scene data.
15 . The agricultural surveillance system of claim 11 , wherein the plugged data includes an indication of a source of a plug.
16 . The agricultural surveillance system of claim 11 , wherein the agricultural machine comprises an implement frame extending laterally between a first side and a second side, the implement frame being configured to support a plurality of ground-engaging tools configured to engage the field,
wherein the plugged data includes an indication of the plugged operating condition being on the first side or the second side.
17 . A method of operating an agricultural surveillance system, the method comprising:
formation flying an autonomous sensor frame based on an agricultural machine;
capturing an image scene with an object detection sensor supported on the autonomous sensor frame and generating scene data representing the image scene, the image scene including at least part of the agricultural machine including a ground-engaging tool;
generating a flight path command for controlling a propulsion system of the autonomous sensor frame based at least in part on a relative orientation of the agricultural machine identified in the scene data;
generating sensor frame data from the autonomous sensor frame that includes plugged data indicative of a plugged operating condition of the ground-engaging tool of the agricultural machine, the plugged data being determined based at least in part on the scene data; and
altering a state of operation of the agricultural machine based on the plugged operating condition of the ground-engaging tool of the agricultural machine.
18 . The method of claim 17 , further comprising, prior to the formation flying, transmitting a global position of the agricultural machine from the agricultural machine to the autonomous sensor frame,
wherein formation flying of the autonomous sensor frame comprises formation flying the autonomous sensor frame based at least in part on the global position of the agricultural machine.
19 . The method of claim 17 , wherein the formation flying includes flying the autonomous sensor frame behind an implement of the agricultural machine.
20 . The method of claim 19 , wherein the scene data includes a residue distribution behind the implement, the plugged data being determined based at least in part on the residue distribution.