Method and apparatus for loader positioning and load handling using vision system
A positioning assist apparatus includes a vision system that generates target device data representative of an obtained image of a set of target devices carried on the boom or tool carrier of the loader or on both, and positioning assist logic executable by a processor to determine a current position and/or pose of the loader based on the target device data, and to generate current loader position data representative of the determined current position of the loader. The loader positon is based on loader height as may be represented by boom angle, and tool carrier angle. Data is generated for display to an operator to visually communicate movement instructions to the operator and/or to generate control signals controlling the loader, each to move the loader from the determined current position to a desired position in response to a return to position (RTP) command received by the apparatus.
1 . A positioning assist apparatus for assisting moving a loader carried on an associated tractor to a desired loader height, the apparatus comprising: a vision system disposed on the associated tractor, the vision system being operable to image a plurality of target devices carried on the loader to generate target device data representative of an image of the plurality of target devices; a loader position control apparatus operatively coupled with the vision system, the loader position control apparatus comprising: a processor device; a non-transient memory device operatively coupled with the processor device and storing the target device data; and loader positioning logic stored in the non-transient memory device; and a display unit operatively coupled with the loader position control apparatus and comprising a screen that is viewable by an operator of the associated tractor, wherein the processor device is operable to execute the loader positioning logic to: determine, based on the target device data, a current loader height of the loader comprising a height of a boom of the loader and a carrier angle of a tool carrier of the loader; and generate, based on the determined current loader height, current loader height data representative of the determined current loader height, wherein the display unit is operable to display on the screen, based on the generated current loader height data, a current loader height image viewable by the operator of the associated tractor, the current loader height image visually communicating the determined current loader height to the operator for assisting moving the loader to the desired loader height.
2 . The positioning assist apparatus according to claim 1 , wherein:
the non-transient memory device stores desired loader height data representative of the desired loader height;
the non-transient memory device stores assistive operation logic; and
the processor device is operable to execute the assistive operation logic to:
determine, based on the desired loader height data and the current loader height data, a difference between the desired loader height and the determined current loader height; and
generate, based on the determined difference between the desired loader height and the determined current loader height, loader movement command data representative of an incremental movement instruction required for effecting movement of the loader from the determined current loader height to the desired loader height,
wherein the display unit is operable to display on the screen, based on the generated loader movement command data, a loader movement instruction image viewable by the operator of the associated tractor for visually communicating to the operator the determined difference between the desired loader height and the determined current loader height for assisting moving the loader to the desired loader height.
3 . The positioning assist apparatus according to claim 2 , further comprising: loader task data stored in the non-transient memory device, the loader task data being representative of a desired operation to be performed by the loader and comprising a first set of loader height command data representative of an ordered sequence of loader heights to assume by the loader to perform the desired operation, wherein the processor device is operable to execute the assistive operation logic to, in turn for each loader height command data of the first set of loader height command data: retrieve a further desired loader height data representative of a further desired loader height; determine, based on the vision system imaging the plurality of target devices, a further current loader height of the loader; generate, based on the determined further current loader height, further current loader height data representative of the determined further current loader height; determine, based on the further desired loader height data and the further current loader height data, a further difference between the further desired loader height and the determined further current loader height; and generate, based on the determined difference between the further desired loader height and the determined further current loader height, further loader movement command data representative of a further incremental movement instruction required for effecting movement of the loader from the determined further current loader height to the further desired loader height, wherein the display unit is operable to display on the screen, based on the generated further loader movement command data, a further loader movement instruction image viewable by the operator of the associated tractor for visually communicating to the operator the determined difference between the further desired loader height and the determined further current loader height for assisting moving the loader to the further desired loader height.
4 . The positioning assist apparatus according to claim 3 , further comprising: training logic stored in the non-transient memory device; and an operator input device operatively coupled with the loader position control apparatus, the operator input device being operable to receive a training confirmation signals, wherein, the processor device is operable to execute the training logic to generate the loader task data during a training session of the positioning assist apparatus by: with the loader disposed in a first desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate first target device training data representative of an image of the plurality of target devices obtained with the loader disposed in the first desired height; and storing, based on the operator input device receiving a first training confirmation signal from the operator of the associated tractor, the first target device training data in the non-transient memory device as a first of the set of loader height command data of the loader task data; with the loader disposed in a second desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate second target device training data representative of an image of the plurality of target devices obtained with the loader disposed in the second desired height; and storing, based on the operator input device receiving a second training confirmation signal from the operator of the associated tractor, the second target device training data in the non-transient memory device as a second of the set of loader height command data of the loader task data.
5 . The positioning assist apparatus according to claim 1 , wherein:
the non-transient memory device stores desired loader height data representative of the desired height of the loader;
the non-transient memory device stores automated operation logic; and
the processor device is operable to execute the automated operation logic to:
determine, based on the desired loader height data and the current loader height data, a difference between the desired loader height and the determined current loader height; and
generate, based on the determined difference between the desired loader height and the determined current loader height, a loader movement control signal representative of an incremental movement instruction required for effecting movement of the loader from the determined current loader height to the desired loader height,
wherein the loader position control apparatus is operable to deliver the loader movement control signal to a controller of the associated tractor to control one or more cylinders of the loader to automatically move the loader from the determined current loader height to the desired loader height in response to the controller receiving the loader movement control signal.
6 . The positioning assist apparatus according to claim 5 , further comprising: loader task data stored in the non-transient memory device, the loader task data being representative of a desired operation to be performed by the loader and comprising a first set of loader height command data representative of an ordered sequence of loader heights to assume by the loader to perform the desired operation, wherein the processor device is operable to execute the automated operation logic to, in turn for each loader height command data of the first set of loader height command data: retrieve a further desired loader height data representative of a further desired loader height; determine, based on the vision system imaging the plurality of target devices, a further current loader height of the loader; generate, based on the determined further current loader height, further current loader height data representative of the determined further current loader height; determine, based on the further desired loader height data and the further current loader height data, a further difference between the further desired loader height and the determined further current loader height; and generate, based on the determined difference between the further desired loader height and the determined further current loader height, a further loader movement control signal representative of a further incremental movement instruction required for effecting movement of the loader from the determined further current loader height to the further desired loader height, wherein the loader position control apparatus is operable to deliver the further loader movement control signal to the controller of the associated tractor to control the one or more cylinders of the loader to automatically move the loader from the determined further current loader height to the further desired loader height in response to the controller receiving the further loader movement control signal.
7 . The positioning assist apparatus according to claim 6 , further comprising: training logic stored in the non-transient memory device; and an operator input device operatively coupled with the loader position control apparatus, the operator input device being operable to receive training confirmation signals, wherein, the processor device is operable to execute the training logic to generate the loader task data during a training session of the positioning assist apparatus by: with the loader disposed in a first desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate first target device training data representative of an image of the plurality of target devices obtained with the loader disposed in the first desired height; and storing, based on the operator input device receiving a first training confirmation signal from the operator of the associated tractor, the first target device training data in the non-transient memory device as a first of the set of loader height command data of the loader task data; with the loader disposed in a second desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate second target device training data representative of the image of the plurality of target devices obtained with the loader disposed in the second desired height; and storing, based on the operator input device receiving a second training confirmation signal from the operator of the associated tractor, the second target device training data in the non-transient memory device as a second of the set of loader height command data of the loader task data.
8 . The positioning assist apparatus according to claim 1 , wherein: the plurality of target devices comprises a set of one or more boom target devices carried on a boom of the loader, and a set of one or more tool carrier target devices carried on a tool carrier of the loader; the vision system is operable to image the set of one or more boom target devices to generate boom target device data representative of the image of the set of one or more boom target devices; the vision system is operable to image the set of one or more tool carrier target devices to generate tool carrier target device data representative of the image of the set of one or more tool carrier target devices; the processor device is operable to execute the loader positioning logic to: determine, based on the boom target device data, the height of the boom of the loader, and generate based on the determined height of the boom current boom height data representative of the determined height of the boom of the loader; and determine, based on the tool carrier target device data, the carrier angle of the tool carrier, and generate, based on the determined carrier angle of the tool carrier, current carrier angle data representative of the determined carrier angle of the tool carrier; and the display unit is operable to display on the screen, based on the generated current boom height and carrier angle data, a tool position image viewable by the operator of the associated tractor for assisting the operator move the tool carrier and boom of the loader to a desired position.
9 . The positioning assist apparatus according to claim 8 , further comprising:
position determination logic stored in the non-transient memory device,
wherein the processor device is operable to execute position determination logic to determine the height of the boom of the loader by:
determining, based on the boom target device data generated by the vision system imaging the set of one or more boom target devices, a location of the set of one or more boom target devices in a coordinate reference framework;
transforming the determined location of the set of one or more boom target devices in the coordinate reference framework into a location of the set of one or more boom target devices in a coordinate reference framework of the associated tractor and vision system operatively coupled with the associated tractor; and
determining, based on the location of the set of one or more boom target devices in the coordinate reference framework of the associated tractor and vision system, a height of the loader relative to ground supporting the associated tractor as the determined height of the boom of the loader.
10 . The positioning assist apparatus according to claim 9 , wherein:
the processor device is operable to execute position determination logic to determine the location of the set of one or more boom target devices in the coordinate reference framework by:
determining a 6-dimensional (6-D) pose of the set of one or more boom target devices.
11 . The positioning assist apparatus according to claim 1 , wherein: the plurality of target devices comprises a set of one or more boom target devices carried on a boom of the loader; the vision system is operable to image the set of one or more boom target devices to generate boom target device data representative of the image of the set of one or more boom target devices; the processor device is operable to execute the loader positioning logic to determine a height of the boom of the loader by: determining a location of the set of one or more boom target devices in a coordinate reference framework, based on the boom target device data generated by the vision system imaging the set of one or more boom target devices by determining a 6-dimensional (6-D) pose of the set of one or more boom target devices; transforming the determined location of the set of one or more boom target devices in the coordinate reference framework into a location of the set of one or more boom target devices in a coordinate reference framework of the associated tractor and vision system operatively coupled with the associated tractor; and determining, based on the location of the set of one or more boom target devices in the coordinate reference framework of the associated tractor and vision system, a height of the loader relative to ground supporting the associated tractor as the determined height of the boom of the loader; and the display unit is operable to display on the screen, based on the generated current boom height data, a tool position image viewable by the operator of the associated tractor for assisting the operator move the boom of the loader to a desired position.
12 . A method for assisting moving a loader carried on an associated tractor to a desired loader height, the method comprising: imaging by a vision system of a positioning assist apparatus and being disposed on the associated tractor a plurality of target devices carried on the loader; generating target device data by the vision system, the target device data being representative of an image of the plurality of target devices; storing loader positioning logic in a non-transient memory device of a loader position control apparatus of the positioning assist apparatus, the loader position control apparatus being operatively coupled with the vision system and comprising a processor device operatively coupled with the non-transient memory device; executing the loader positioning logic by the processor device to: determine, based on the target device data, a current loader height of the loader comprising a height of a boom of the loader and a carrier angle of a tool carrier of the loader; and generate, based on the determined current loader height, current loader height data representative of the determined current loader height, displaying based on the generated loader height data a current loader height image on a screen of a display unit operatively coupled with the loader position control apparatus, wherein the current loader height image displayed on the screen is viewable by an operator of the associated tractor, wherein the current loader height image visually communicates the determined current loader height to the operator for assisting moving the loader to the desired loader height.
13 . The method according to claim 12 , further comprising:
storing assistive operation logic and desired loader height data in the non-transient memory device, the desired loader height data being representative of the desired loader height;
executing the assistive operation logic by the processor device to:
determine, based on the desired loader height data and the current loader height data, a difference between the desired loader height and the determined current loader height; and
generate, based on the determined difference between the desired loader height and the determined current loader height, loader movement command data representative of an incremental movement instruction required for effecting movement of the loader from the determined current loader height to the desired loader height; and
displaying on the screen of the display unit based on the generated loader movement command data a loader movement instruction image viewable by the operator of the associated tractor or visually communicating to the operator the determined difference between the desired loader height and the determined current loader height for assisting moving the loader to the desired loader height.
14 . The method according to claim 13 , further comprising:
storing loader task data in the non-transient memory device, the loader task data being representative of a desired operation to be performed by the loader and comprising a first set of loader height command data representative of an ordered sequence of loader heights to assume by the loader to perform the desired operation;
executing the assistive operation logic by the processor device to, in turn for each loader height command data of the first set of loader height command data:
retrieve a further desired loader height data representative of a further desired loader height;
determine, based on the vision system imaging the target device data, a further current loader height of the loader;
generate, based on the determined further current loader height, further current loader height data representative of the determined further current loader height;
determine, based on the further desired loader height data and the further current loader height data, a further difference between the further desired loader height and the determined further current loader height; and
generate, based on the determined difference between the further desired loader height and the determined further current loader height, further loader movement command data representative of a further incremental movement instruction required for effecting movement of the loader from the determined further current loader height to the further desired loader height; and
displaying on the screen of the display unit based on the generated further loader movement command data a further loader movement instruction image viewable by the operator of the associated tractor for visually communicating to the operator the determined difference between the further desired loader height and the determined further current loader height for assisting moving the loader to the further desired loader height.
15 . The method according to claim 14 , further comprising: storing training logic stored in the non-transient memory device; executing the training logic by the processor device to generate the loader task data during a training session of the positioning assist apparatus by: with the loader disposed in a first desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate first target device training data representative of the image of the plurality of target devices obtained with the loader disposed in the first desired height; and storing, based on receiving a first training confirmation signal from the operator of the associated tractor via an operator input device operatively coupled with the loader position control apparatus, the first target device training data in the non-transient memory device as a first of the set of loader height command data of the loader task data; with the loader disposed in a second desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate second target device training data representative of the image of the plurality of target devices obtained with the loader disposed in the second desired height; and storing, based on receiving a second training confirmation signal from the operator of the associated tractor via the operator input device operatively coupled with the loader position control apparatus, the second target device training data in the non-transient memory device as a second of the set of loader height command data of the loader task data.
16 . The method according to claim 12 , further comprising:
storing desired loader height data in the non-transient memory device, the desired loader height data being representative of the desired loader height;
storing automated operation logic in the non-transient memory device;
executing the automated operation logic by the processor device to:
determine, based on the desired loader height data and the current loader height data, a difference between the desired loader height and the determined current loader height; and
generate, based on the determined difference between the desired loader height and the determined current loader position, a loader movement control signal representative of an incremental movement instruction required for effecting movement of the loader from the determined current loader height to the desired loader height; and
delivering the loader movement control signal to a controller of the associated tractor to control one or more cylinders of the loader to automatically move the loader from the determined current loader height to the desired loader height in response to the controller receiving the loader movement control signal.
17 . The method according to claim 16 , further comprising:
storing loader task data in the non-transient memory device, the loader task data being representative of a desired operation to be performed by the loader and comprising a first set of loader height command data representative of an ordered sequence of loader heights to assume by the loader to perform the desired operation;
executing the automated operation logic by the processor device to, in turn for each loader height command data of the first set of loader height command data:
retrieve a further desired loader height data representative of a further desired loader height;
determine, based on the vision system imaging the target device data, a further current loader height of the loader;
generate, based on the determined further current loader height, further current loader height data representative of the determined further current loader height;
determine, based on the further desired loader height data and the further current loader height data, a further difference between the further desired loader height and the determined further current loader height; and
generate, based on the determined difference between the further desired loader height and the determined further current loader height, a further loader movement control signal representative of a further incremental movement instruction required for effecting movement of the loader from the determined further current loader height to the further desired loader height; and
delivering the further loader movement control signal to the controller of the associated tractor to control the one or more cylinders of the loader to automatically move the loader from the determined further current loader height to the further desired loader height in response to the controller receiving the further loader movement control signal.
18 . The method according to claim 17 , further comprising: storing training logic in the non-transient memory device; executing the training logic by the processor device to generate the loader task data during a training session of the positioning assist apparatus by: with the loader disposed in a first desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate first target device training data representative of the image of the plurality of target devices obtained with the loader disposed in the first desired height; and storing, based on receiving a first training confirmation signal from the operator of the associated tractor via an operator input device operatively coupled with the loader position control apparatus, the first target device training data in the non-transient memory device as a first of the set of loader height command data of the loader task data; with the loader disposed in a second desired height of the ordered sequence of loader heights to assume by the loader to perform the desired operation: imaging by the vision system the plurality of target devices carried on the loader to generate second target device training data representative of the image of the plurality of target devices obtained with the loader disposed in the second desired height; and storing, based on receiving a second training confirmation signal from the operator of the associated tractor via the operator input device operatively coupled with the loader position control apparatus, the second target device training data in the non-transient memory device as a second of the set of loader height command data of the loader task data.
19 . The method according to claim 12 , further comprising: storing position determination logic in the non-transient memory device; imaging by the vision system a set of one or more boom target devices carried on a boom of the loader to generate boom target device data representative of the image of the set of one or more boom target devices; imaging by the vision system a set of one or more tool carrier target devices carried on a tool carrier of the loader to generate tool carrier target device data representative of the image of the set of one or more tool carrier target devices; executing the position determination logic by the processor device to: determine, based on the boom target device data, the height of the boom of the loader, and generate based on the determined height of the boom current boom height data representative of the determined height of the boom of the loader; determine, based on the tool carrier target device data, the carrier angle of the tool carrier, and generate, based on the determined carrier angle of the tool carrier, current carrier angle data representative of the determined carrier angle of the tool carrier; determine, based on the boom target device data generated by the vision system imaging the set of one or more boom target devices, a location of the set of one or more boom target devices in a coordinate reference framework by determining a 6-dimensional (6-D) pose of the set of one or more boom target devices; transform the determined location of the set of one or more boom target devices in the coordinate reference framework into a location of the set of one or more boom target devices in a coordinate reference framework of the associated tractor and vision system operatively coupled with the associated tractor; determine, based on the location of the set of one or more boom target devices in the coordinate reference framework of the associated tractor and vision system, a height of the loader relative to ground supporting the associated tractor as the determined height of the boom of the loader; and displaying on the screen of the display unit based on the generated current boom height and carrier angle data, a tool position image viewable by the operator of the associated tractor for assisting the operator move the tool carrier and boom of the loader to a desired position.
20 . The method according to claim 12 , further comprising:
imaging by the vision system a set of one or more boom target devices carried on a boom of the loader to generate boom target device data representative of the image of the set of one or more boom target devices;
executing the loader positioning logic by the processor device to determine a height of the boom of the loader by:
determining a location of the set of one or more boom target devices in a coordinate reference framework, based on the boom target device data generated by the vision system imaging the set of one or more boom target devices by determining a 6-dimensional (6-D) pose of the set of one or more boom target devices;
transforming the determined location of the set of one or more boom target devices in the coordinate reference framework into a location of the set of one or more boom target devices in a coordinate reference framework of the associated tractor and vision system operatively coupled with the associated tractor; and
determining, based on the location of the set of one or more boom target devices in the coordinate reference framework of the associated tractor and vision system, a height of the loader relative to ground supporting the associated tractor as the determined height of the boom of the loader; and
displaying on the screen of the display unit based on the generated current boom height data, a tool position image viewable by the operator of the associated tractor for assisting the operator move a tool carrier and boom of the loader to a desired position.