Method for estimating operation of work vehicle, system, method for producing trained classification model, training data, and method for producing training data

1. A method performed by a computer, the method comprising:

obtaining motion data indicating a motion change of a work vehicle, the motion data being generated from a plurality of still images indicating the work vehicle performing a prescribed operation in time series, the plurality of still images showing the work vehicle in a plurality of different postures during performance of the prescribed operation, the motion data being a still image generated by superimposing or connecting the plurality of still images; and

determining an operation classification of the work vehicle from the motion data by performing image classification using a trained classification model, the operation classification corresponding to the prescribed operation performed by the work vehicle, the operation classification being one of a plurality of classifications, each of the plurality of classifications corresponding to a different operation performed by the work vehicle and being determinable from the motion data.

2. The method according to claim 1 , wherein

the motion data is obtained by combining pixel values of the plurality still images.

3. The method according to claim 1 , wherein

the motion data indicates a group of images included in the plurality of still images.

4. The method according to claim 1 , wherein

the motion data is represented by an average value of pixel values corresponding to each other in the plurality of still images,

the motion data is obtained by calculating an average value of pixel values of corresponding pixels in each of the plurality still images.

5. The method according to claim 1 , further comprising:

obtaining a video of the work vehicle in operation;

extracting the plurality of still images from the video; and

generating the motion data from the extracted plurality of still images.

6. The method according to claim 5 , wherein

the extracting the plurality of still images includes extracting the images by skipping a part of consecutive frames included in the video.

7. The method according to claim 5 , wherein

the generating the motion data includes changing a portion of the plurality of still images to a darkness different from the other images.

8. The method according to claim 1 , wherein

the generating the motion data includes gray scaling the plurality of still images.

9. The method according to claim 1 , further comprising:

recording an operation time of the work vehicle indicated by the classification; and

generating management data including the classification and the operation time.

10. The method according to claim 1 , wherein

the classification model includes a neural network.

11. The method according to claim 10 , wherein

the neural network includes an input layer and an output layer, the input layer having a number of neurons corresponding to a number of pixels in the plurality of still images and the output layer having a number of neurons corresponding to a number of the plurality of classifications.

12. The method according to claim 1 , wherein

the plurality of classifications corresponds to operations including at least one of turning, discharging, and excavating.

13. The method according to claim 1 , wherein

the motion data is obtained by calculating an average value of pixel values of corresponding pixels in each of the plurality still images,

a plurality of output values is obtained, each of the output values being a probability that the motion data indicates each of the plurality of classifications, respectively, and

the operation classification is determined by determining which of the output values is highest.

14. The method according to claim 1 , further comprising

determining a plurality of output values, each of the output values being a probability that the motion data indicates each of the plurality of classifications, respectively, and

the operation classification is determined by determining which of the output values is highest.

15. A system comprising:

a trained classification model; and

a processor programmed to

obtain motion data indicating a motion change of a work vehicle, the motion data being generated from a plurality of still images showing the work vehicle performing a prescribed operation in time series, the plurality of still images showing the work vehicle in a plurality of different postures during performance of the prescribed operation, the motion data being a still image generated by superimposing or connecting the plurality of still images, and

determine an operation classification of the work vehicle from the motion data by performing image classification using the trained classification model, the operation classification corresponding to the prescribed operation performed by the work vehicle, the operation classification being one of a plurality of classifications, each of the plurality of classifications corresponding to a different operation performed by the work vehicle and being determinable from the motion data.

16. The system according to claim 15 , further comprising:

a camera configured to shoot a video of the work vehicle, and

the processor programmed to

obtain the video from the camera,

extract the plurality of still images from the video, and

generate the motion data from the plurality of extracted images.

17. The system according to claim 15 , wherein

the processor is further programmed to

record an operation time of the work vehicle indicated by the classification, and

generate management data including the classification and the operation time.

18. The system according to claim 15 , wherein

the classification model includes a neural network.

19. A producing method for a trained classification model, the producing method comprising:

obtaining training data including

motion data indicating a motion change of a work vehicle, the motion data being generated from a plurality of still images indicating the work vehicle performing a prescribed operation in time series, the plurality of still images showing the work vehicle in a plurality of different postures during performance of the prescribed operation, the motion data indicating a still image generated by superimposing or connecting the plurality of still images, and

an operation classification of the work vehicle assigned to the motion data, the operation classification corresponding to the prescribed operation performed by the work vehicle, the operation classification being one of a plurality of classifications, each of the plurality of classifications corresponding to a different operation performed by the work vehicle and being determinable from the motion data; and

training a classification model with the training data.

20. The producing method according to claim 19 , further comprising:

obtaining a video of the work vehicle in operation;

extracting the plurality of still images from the video;

generating the motion data from the plurality of still images; and

assigning the operation classification of the work vehicle to the motion data.

21. The method according to claim 20 , wherein

the classification model is configured to generate output values, each of the output values indicating a probability that the motion data indicates each of the plurality of classifications, respectively, and

the determining the operation classification of the work vehicle includes determining which of the output values is highest.

22. A training data for training a classification model, the training data comprising:

motion data indicating a motion change of a work vehicle, the motion data being generated from a plurality of still images indicating the work vehicle performing a prescribed operation in time series, the plurality of still images showing the work vehicle in a plurality of different postures during performance of the prescribed operation, the motion data being a still image generated by superimposing or connecting the plurality of still images; and

an operation classification of the work vehicle assigned to the motion data, the operation classification corresponding to the prescribed operation performed by the work vehicle, the operation classification being one of a plurality of classifications, each of the plurality of classifications corresponding to a different operation performed by the work vehicle and being determinable from the motion data.

23. The training data according to claim 22 , wherein

the still image is generated by superimposing the plurality of still images, and

the superimposed image includes

an original image, and

a plurality of processed images obtained by performing one or more of reduction, enlargement, rotation, translation, left-right reversal, and color change on the original image.

24. A producing method for training data to training a classification model, the producing method comprising:

obtaining a plurality of still images showing a work vehicle performing a prescribed operation in time series, the plurality of still images showing the work vehicle in a plurality of different postures during performance of the prescribed operation;

generating motion data indicating a motion change of the work vehicle from the plurality of still images, the motion data being a still image generated by superimposing or connecting the plurality of still images; and

obtaining an operation classification of the work vehicle assigned to the motion data, the operation classification corresponding to the prescribed operation performed by the work vehicle, the operation classification being one of a plurality of classifications, each of the plurality of classifications corresponding to a different operation performed by the work vehicle and being determinable from the motion data.

25. The producing method according to claim 24 , further comprising:

obtaining a video of the work vehicle,

the obtaining the plurality of still images including extracting the plurality of still images from the video.

26. A producing method for a trained classification model, the producing method comprising:

obtaining motion data indicating a motion change of a work vehicle, the motion data being generated from a plurality of still images indicating the work vehicle performing a prescribed operation in time series, the plurality of still images showing the work vehicle in a plurality of different postures during performance of the prescribed operation, the motion data being a still image generated by superimposing or connecting the plurality of still images;

determining an operation classification of the work vehicle from the motion data by performing image classification using a trained first classification model, the operation classification corresponding to the prescribed operation performed by the work vehicle, the operation classification being one of a plurality of classifications, each of the plurality of classifications corresponding to a different operation performed by the work vehicle and being determinable from the motion data; and

training a second classification model with training data including the motion data and the determined operation classification of the work vehicle.