Activity detection device, activity detection system, and activity detection method
An object of the disclosure is to provide flexible and highly accurate activity detection means. Provided is an activity detection device including: an input unit that inputs an image sequence including a first image and a second image; an object detection unit that detects a first object in the image sequence; a component model unit that generates first characteristic information characterizing the first object and includes at least one individually trainable component model; and an activity detection unit that generates a first object state corresponding to the first object in the first image and a second object state corresponding to the first object in the second image based on the first characteristic information and determines an activity related to the first object based on the first and second object states.
1. An activity detection device comprising:
a processor; and
a memory coupled to the processor, the memory storing instructions that when executed by the processor configures the processor to:
store a plurality of individually trainable component models which are machine learning models including at least a convolutional neural network model and an object skeleton model,
input an image sequence including at least a first image and a second image,
detect a first object in the image sequence,
generate first characteristic information characterizing the first object, which includes a first feature quantity of the first object generated by the convolutional neural network and first skeleton information of the first object generated by the object skeleton model,
generate a first object state corresponding to the first object in the first image and a second object state corresponding to the first object in the second image based on the first feature quantity of the first object and the first skeleton information of the first object as the first characteristic information,
determine an activity related to the first object based on the first object state and the second object state, and
output a first activity detection result indicating the determined activity, and
wherein the processor is further configured to:
detect a second object in the image sequence,
determine second characteristic information characterizing the second object which includes a second feature quantity of the second object generated by the convolutional neural network and second skeleton information of the second object generated by the object skeleton model, and
generate a third object state corresponding to the second object in the first image based on the second characteristic information,
determine an activity related to the first object based on the first object state, the second object state, and the third object state, and
output a second activity detection result indicating the determined activity.
2. The activity detection device according to claim 1 ,
wherein the plurality of component models includes at least one of an optical flow model, an object tracking model, an object re-identification model, and an object attribute model.
3. The activity detection device according to claim 1 ,
wherein each component model generates information related to a generation process of the first characteristic information as an explanation notification, and
wherein the processor is configured to output the explanation notification together with the first activity detection result.
4. The activity detection device according to claim 3 ,
wherein the first object state and the second object state indicate a characteristic characterizing the first object in a semantic binary format.
5. The activity detection device according to claim 4 ,
wherein the processor is configured to generate the first object state and the second object state in a Markov model, and determine the activity related to the first object based on a relationship between the first object state and the second object state in the Markov model.
6. The activity detection device according to claim 5 ,
wherein the processor is configured to determine the activity related to the first object based on a transition from the first object state to the second object state in the Markov model.
7. The activity detection device according to claim 6 ,
wherein the processor is configured to derive an object state corresponding to an event not shown in the image sequence based on a causal relationship between the first object state and the second object state in the Markov model.
8. The activity detection device according to claim 7 ,
wherein the processor is configured to output the relationship between the first object state and the second object state used in the determination of the activity related to the first object as the explanation notification.
9. The activity detection system according to claim 1 ,
wherein the first characteristic information is information indicating characteristics of the first object including at least one of spatial coordinates, temporal coordinates, a moving path, a size, or a color.
10. An activity detection system in which a client terminal and an activity detection device are connected to each other via a communication network,
wherein the activity detection device comprises:
a processor; and
a memory coupled to the processor, the memory storing instructions that when executed by the processor configures the processor to:
store a plurality of individually trainable component models which are machine learning models including at least a convolutional neural network model and an object skeleton model,
input an image sequence including at least a first image and a second image from the client terminal,
detect a first object in the image sequence,
generate first characteristic information characterizing the first object which includes a first feature quantity of the first object generated by the convolutional neural network and first skeleton information of the first object generated by the object skeleton model,
generate a first object state corresponding to the first object in the first image and a second object state corresponding to the first object in the second image based on the first feature quantity of the first object and the first skeleton information of the first object as the first characteristic information,
determine an activity related to the first object based on the first object state and the second object state, and
output an activity detection result indicating the determined activity to the client terminal, and
wherein the processor is further configured to:
detect a second object in the image sequence,
determine second characteristic information characterizing the second object which includes a second feature quantity of the second object generated by the convolutional neural network and second skeleton information of the second object generated by the object skeleton model,
generate a third object state corresponding to the second object in the first image based on the second characteristic information,
determine an activity detection result related to the first object based on the first object state, the second object state, and the third object state, and
output a second activity detection result indicating the determined activity.
11. The activity detection system according to claim 10 ,
wherein the first characteristic information is information indicating characteristics of the first object including at least one of spatial coordinates, temporal coordinates, a moving path, a size, or a color.
12. An activity detection method comprising:
a step of storing a plurality of individually trainable component models which are machine learning models including at least a convolutional neural network model and an object skeleton model;
a step of inputting an image sequence including at least a first image and a second image;
a step of detecting a first object in the image sequence;
a step of generating first characteristic information characterizing the first object which includes a first feature quantity of the first object generated by the convolutional neural network and first skeleton information of the first object generated by the object skeleton model; and
a step of generating a first object state corresponding to the first object in the first image and a second object state corresponding to the first object in the second image based on the first feature quantity of the first object and the first skeleton information of the first object as the first characteristic information;
determining an activity related to the first object based on the first object state and the second object state;
outputting an activity detection result indicating the determined activity,
a step of detecting a second object in the image sequence;
a step of generating second characteristic information characterizing the second object which includes a second feature quantity of the second object generated by the convolutional neural network and second skeleton information of the second object generated by the object skeleton model; and
a step of generating a third object state corresponding to the second object in the first image based on the second characteristic information;
a step of determining an activity related to the first object based on the first object state, the second object state, and the third object state; and
a step of outputting a second activity detection result indicating the determined activity.
13. The activity detection method according to claim 12 ,
wherein the step of generating first characteristic information characterizing the first object by using at least one individually trainable component model further comprises a step of generating information related to a generation process of the first characteristic information as an explanation notification.
14. The activity detection method according to claim 12 ,
wherein the first characteristic information is information indicating characteristics of the first object including at least one of spatial coordinates, temporal coordinates, a moving path, a size, or a color.