Ground engaging tool wear and loss detection system and method
A wear detection system can be configured to receive a video stream including a plurality of images of a bucket of the work machine from a camera associated with the work machine. The bucket has one or more ground engaging tools (GET). The wear detection system can also be configured to identify a plurality of tool images from the video stream over a period of time. The plurality of tool images depict the GET at a plurality of instances over a period of time. The wear detection system can also be configured to determine a plurality of tool pixel counts from the plurality of tool image and determine a wear level for the GET based on the plurality of tool pixel counts.
1. A method comprising:
receiving from a camera associated with a work machine, a video stream including a plurality of images of a bucket of the work machine, the bucket having a model number and at least one ground engaging tool (GET);
based on the model number, selecting:
a matching bucket-tool template corresponding to the bucket from a plurality of bucket-tool templates stored in a template library of the work machine,
a predetermined moment in a dig-dump cycle of the work machine, and
a predetermined position of the bucket during the predetermined moment in the dig-dump cycle;
identifying a plurality of tool images from the video stream over a period of time that matches one or more stored images from the matching bucket-tool template, the plurality of tool images capturing the bucket at the predetermined position during the predetermined moment in the dig-dump cycle and depicting the at least one GET at a plurality of time instances over the period of time;
determining a plurality of tool pixel counts from the plurality of tool images based at least in part on a shape and style of the GET, the plurality of tool pixel counts indicating at least one of:
an area of the GET,
a height of the GET,
a width of the GET, or
a sum of the height and the width of the GET; and
determining a wear level or loss for the at least one GET based on the plurality of tool pixel counts.
2. The method of claim 1 further including:
generating an alert based on the wear level or loss including the wear level reaching a wear threshold value; and
displaying the alert on a display of an operator control panel of the work machine, the alert including at least one of:
the at least GET being lost,
the at least GET requiring a replacement, or
a replacement time for the at least GET.
3. The method of claim 1 wherein determining the wear level or loss for the at least one GET based on the plurality of tool pixel counts comprises mapping the plurality of time instances over the period of time to the plurality of tool pixel counts.
4. The method of claim 1 further comprising determining a wear-level trend.
5. The method of claim 4 further comprising predicting a replacement time for the at least one GET based on the wear-level trend.
6. The method of claim 4 wherein the wear-level trend is determined by mapping the plurality of time instances to the plurality of tool pixel counts.
7. The method of claim 1 wherein identifying the plurality of tool images from the video stream over the period of time includes:
performing a segmentation analysis on the plurality of tool images, and
finding one or more images from the plurality of tool images that match the one or more stored images.
8. The method of claim 1 wherein identifying the plurality of tool images from the video stream comprises selecting a plurality of video frames from the video stream based on a segmentation analysis applied to the video stream.
9. The method of claim 1 , wherein:
the predetermined moment in the dig-dump cycle includes:
a dig end of the dig-dump cycle,
a midpoint in the dig-dump cycle, or
a dump end of the dig-dump cycle; and
the predetermined position of the bucket during the predetermined moment includes:
the bucket at rest at the dig end of the dig-dump cycle,
the bucket within a range of positions at the midpoint in the dig-dump cycle, or
the bucket at rest at the dump end of the dig-dump cycle.
10. A system comprising:
a camera;
a processor; and
a non-transitory computer readable medium storing executable instructions that, when executed by the processor, cause the processor to perform operations, the operations comprising:
receiving, from the camera, a video stream including a plurality of images of a bucket of a work machine, the bucket having a model number and at least one ground engaging tool (GET);
based on the model number, selecting:
a matching bucket-tool template corresponding to the bucket from a plurality of bucket-tool templates stored in a template library of the non-transitory computer readable medium,
a predetermined moment in a dig-dump cycle of the work machine, and
a predetermined position of the bucket during the predetermined moment in the dig-dump cycle;
identifying a plurality of tool images from the video stream that matches one or more stored images from the matching bucket-tool template, the plurality of tool images capturing the bucket at the predetermined position during the predetermined moment in the dig-dump cycle and depicting the GET at a plurality of time instances over a period of time;
determining a plurality of tool pixel counts from the plurality of tool images based at least in part on a shape and style of the GET, the plurality of tool pixel counts indicating at least one of:
an area of the GET,
a height of the GET,
a width of the GET, or
a sum of the height and the width of the GET; and
determining a wear level or loss for the at least one GET based on the plurality of tool pixel counts.
11. The system of claim 10 wherein the operations further comprise generating an alert based on the wear level or loss.
12. The system of claim 10 wherein determining the wear level or loss for the at least one GET based on the plurality of tool pixel counts comprises mapping the plurality of time instances over the period of time to the plurality of tool pixel counts.
13. The system of claim 10 wherein the operations further comprise determining a wear-level trend.
14. The system of claim 13 wherein the operations further comprise predicting a replacement time for the at least one GET based on the wear-level trend.
15. The system of claim 13 wherein the wear-level trend is determined by mapping the plurality of time instances to the plurality of tool pixel counts.
16. The system of claim 10 wherein identifying the plurality of tool images from the video stream over the period of time includes:
performing a segmentation analysis on the plurality of tool images, and
finding one or more images from the plurality of tool images that match the one or more stored images.
17. The system of claim 10 wherein identifying the plurality of tool images includes:
calculating a similarity score for the plurality of tool images and the one or more stored images, wherein the similarity score reflects a measure of the plurality of tool images matching the one or more stored images.
18. The system of claim 10 wherein identifying the plurality of tool images includes:
calculating a similarity score for the plurality of tool images, wherein the similarity score indicates the wear level of the plurality of tool images.
19. A system comprising:
a camera configured to capture a video stream of a bucket having a model number and a plurality of ground engaging tools (GET) during a moment in a dig-dump cycle of a work machine when the plurality of GET is within a field of view of the camera;
a processor; and
a computer readable medium storing executable instructions that, when executed by the processor, cause the processor to perform operations, the operations comprising:
receiving the video stream from the camera, the video stream including a plurality of images of the bucket;
based on the model number, selecting:
a matching bucket-tool template corresponding to the bucket from a plurality of bucket-tool templates stored in a template library of the work machine,
a predetermined moment in the dig-dump cycle of the work machine, and
a predetermined position of the bucket during the predetermined moment in the dig-dump cycle;
identifying a plurality of tool images from the video stream that match one or more stored images from the matching bucket-tool template, the plurality of tool images capturing the bucket at the predetermined position during the predetermined moment in the dig-dump cycle and depicting, at a plurality of time instances over a period of time, at least one of the plurality of GET;
determining a plurality of tool pixel counts associated with the at least one of the plurality of GET from the plurality of tool images based at least in part on a shape and style of the at least one of the plurality of GET, the plurality of tool pixel counts indicating at least one of:
an area of the at least one of the plurality of GET,
a height of the at least one of the plurality of GET,
a width of the at least one of the plurality of GET, or
a sum of the height and the width of the at least one of the plurality of GET; and
determining a wear level or loss for the at least one of the plurality of GET based on mapping the plurality of time instances to the plurality of tool pixel counts.
20. The system of claim 19 wherein the operations further comprise:
determining a wear-level trend based on the mapping;
predicting a replacement time for the at least one of the plurality of GET based on the wear-level trend;
generating an alert based on the wear level or loss; and
displaying the alert on a display of an operator control panel of the work machine, the alert including at least one of:
the at least one of the plurality of GET being lost,
the at least one of the plurality of GET requiring a replacement, or
a replacement time for the at least one of the plurality of GET.