Movement state recognition model training device, movement state recognition device, methods and programs therefor

1. A movement state recognition model training device configured to train a deep neural network (DNN) model that is input with a time series of image data of a camera mounted to a moving body and a time series of sensor data of a sensor mounted to the moving body, that extracts respective image data features and respective sensor data features, and that recognizes a movement state of the moving body from data abstracted from the respective image data features and the respective sensor data features, the movement state recognition model training device comprising:

a memory; and

a processor that is coupled to the memory and that is configured to:

generate first annotation data indicating whether or not the movement state of the moving body corresponds to any of a plurality of predetermined movement state classes, second annotation data indicating which of the plurality of predetermined movement state classes corresponds to the movement state of the moving body, and third annotation data indicating which of the plurality of predetermined movement state classes and a miscellaneous-other movement state class corresponds to the movement state of the moving body, based on annotation data indicating a pre-appended movement state for the image data time series and the sensor data time series; and

train a parameter of the DNN model based on the image data time series and the sensor data time series and on the first annotation data, the second annotation data, and the third annotation data generated for the image data time series and the sensor data time series, by training the DNN model such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.

2. The movement state recognition model training device of claim 1 , wherein:

the DNN model includes an output layer to output a recognition result indicating whether or not the movement state of the moving body corresponds to one of the plurality of movement state classes, an output layer to output a recognition result indicating which of the plurality of movement state classes corresponds to the movement state of the moving body, and an output layer to output a recognition result indicating which of the plurality of predetermined movement state classes and the miscellaneous-other movement state class corresponds to the movement state of the moving body; and

the processor is further configured to train the parameter of the DNN model such that a recognition result output by each output layer of the DNN model matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.

3. A movement state recognition device comprising:

a memory; and

a processor that is coupled to the memory and that is configured to:

recognize a movement state of a moving body of a recognition subject by inputting a time series of image data of a camera mounted to the moving body and a time series of sensor data of a sensor mounted to the moving body into a deep neural network (DNN) model that has been pre-trained to use the image data time series and the sensor data time series as input, to extract respective image data features and respective sensor data features, and to recognize a movement state of the moving body from data abstracted from the respective image data features and the respective sensor data features,

wherein the DNN model is pre-trained based on first annotation data indicating whether or not the movement state of the moving body corresponds to any of a plurality of predetermined movement state classes, second annotation data indicating which of the plurality of predetermined movement state classes corresponds to the movement state of the moving body, and third annotation data indicating which of the plurality of predetermined movement state classes and a miscellaneous-other movement state class corresponds to the movement state of the moving body, that are generated from annotation data indicating a pre-appended movement state for the image data time series and the sensor data time series, and based on the image data time series and the sensor data time series, by training the DNN model such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.

4. A movement state recognition model training method for training a deep neural network (DNN) model that is input with a time series of image data of a camera mounted to a moving body and a time series of sensor data of a sensor mounted to the moving body, that extracts respective image data features and respective sensor data features, and that recognizes a movement state of the moving body from data abstracted from the respective image data features and the respective sensor data features, the movement state recognition model training method comprising:

by a computer,

generating first annotation data indicating whether or not the movement state of the moving body corresponds to any of a plurality of predetermined movement state classes, second annotation data indicating which of the plurality of predetermined movement state classes corresponds to the movement state of the moving body, and third annotation data indicating which of the plurality of predetermined movement state classes and a miscellaneous-other movement state class corresponds to the movement state of the moving body, based on annotation data indicating a pre-appended movement state for the image data time series and the sensor data time series; and

training a parameter of the DNN model based on the image data time series and the sensor data time series and on the first annotation data, the second annotation data, and the third annotation data generated for the image data time series and the sensor data time series, by training the DNN model such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.

5. A movement state recognition method for recognizing a movement state of a moving body of a recognition subject by a computer inputting a time series of image data of a camera mounted to the moving body and a time series of sensor data of a sensor mounted to the moving body into a deep neural network (DNN) model that has been pre-trained to use the image data time series and the sensor data time series as input, to extract respective image data features and respective sensor data features, and to recognize a movement state of the moving body from data abstracted from the respective image data features and the respective sensor data features, the movement state recognition method comprising:

pre-training the DNN model based on first annotation data indicating whether or not the movement state of the moving body corresponds to any of a plurality of predetermined movement state classes, second annotation data indicating which of the plurality of predetermined movement state classes corresponds to the movement state of the moving body, and third annotation data indicating which of the plurality of predetermined movement state classes and a miscellaneous-other movement state class corresponds to the movement state of the moving body, that are generated from annotation data indicating a pre-appended movement state for the image data time series and the sensor data time series, and based on the image data time series and the sensor data time series, by training the DNN model such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.

6. A non-transitory computer readable medium storing a program executable by a computer to perform a process for training a deep neural network (DNN) model that is input with a time series of image data of a camera mounted to a moving body and a time series of sensor data of a sensor mounted to the moving body, that extracts respective image data features and respective sensor data features, and that recognizes a movement state of the moving body from data abstracted from the respective image data features and the respective sensor data features, the process comprising:

generating first annotation data indicating whether or not the movement state of the moving body corresponds to any of a plurality of predetermined movement state classes, second annotation data indicating which of the plurality of predetermined movement state classes corresponds to the movement state of the moving body, and third annotation data indicating which of the plurality of predetermined movement state classes and a miscellaneous-other movement state class corresponds to the movement state of the moving body, based on annotation data indicating a pre-appended movement state for the image data time series and the sensor data time series; and

training a parameter of the DNN model based on the image data time series and the sensor data time series and on the first annotation data, the second annotation data, and the third annotation data generated for the image data time series and the sensor data time series, by training the DNN model such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.

7. A non-transitory computer readable medium storing a program executable by a computer to perform a process for movement state recognition processing, the process comprising:

recognizing a movement state of a moving body of a recognition subject by inputting a time series of image data of a camera mounted to the moving body and a time series of sensor data of a sensor mounted to the moving body into a deep neural network (DNN) model that has been pre-trained to use the image data time series and the sensor data time series as input, to extract respective image data features and respective sensor data features, and to recognize a movement state of the moving body from data abstracted from the respective image data features and the respective sensor data features,

wherein the DNN model is pre-trained based on first annotation data indicating whether or not the movement state of the moving body corresponds to any of a plurality of predetermined movement state classes, second annotation data indicating which of the plurality of predetermined movement state classes corresponds to the movement state of the moving body, and third annotation data indicating which of the plurality of predetermined movement state classes and a miscellaneous-other movement state class corresponds to the movement state of the moving body, that are generated from annotation data indicating a pre-appended movement state for the image data time series and the sensor data time series, and based on the image data time series and the sensor data time series, by training the DNN model such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data.