Methods for real-time skill assessment of multi-step tasks performed by hand movements using a video camera
Example implementations described herein are directed to systems and methods for skill assessment, such as hand washing compliance in hospitals, or assembling products in factories. Example implementations involve body part tracking (e.g., hands), skeleton tracking and deep neural networks to detect and recognize sub-tasks and to assess the skill on each sub-task. Furthermore, the order of the sub-tasks is checked for correctness. Beyond monitoring individual users, example implementations can be used for analyzing and improving workflow designs with multiple sub-tasks.
1. A method, comprising:
analyzing a plurality of image frames, received from a camera system, using computer vision techniques to detect, in the plurality of image frames, a body part of a user associated with a task to be completed;
grouping sequential image frames of the plurality of image frames received from the camera system into a plurality of sub-tasks of the task to be completed based on an application of a machine learning model configured to associate each of the plurality of subtasks to the sequential image frames, said grouping comprises:
computing a dense optical flow over the plurality of image frames, computing the dense optical flow comprises producing x-component and a y-component images for each of the plurality of image frames;
associating each of the plurality of sub-tasks to sequential image frames through assigning sub-task labels to each of the sequential image frames based on using the machine learning model to classify each of the x-component and y-component of the sequential image frames using the machine learning model,
segmenting out each of the plurality of sub-tasks based on grouping the sub-task labels of the sequential image frames, and
grouping sequential ones of the segmented plurality of sub-tasks as the task to be completed; and
providing an evaluation of completion of the task based on detection of completion for the plurality of sub-tasks and order correctness of the sub-tasks, said evaluation comprises:
identifying an order of correctness of the sub-tasks from the grouping of sequential ones of the plurality of sub-tasks,
identifying completion for each of the plurality of sub-tasks through an application of a computer vision algorithm on corresponding ones of the labeled sequential image frames, and
computing a skill score for the task to be completed based on the identified order of correctness, the identified completion of the each of the plurality of sub-tasks, and a skill score for the each of the plurality of sub-tasks.
2. The method of claim 1 , wherein the detecting from the camera system, the body part of the user associated with the task to be completed comprises extracting a frame region around the body part for each image frame of the camera system.
3. The method of claim 1 , wherein the body part of the user comprises hands of the user, wherein the task to be completed is hand washing, wherein the plurality of sub-tasks comprises steps for conducting the hand washing.
4. The method of claim 1 , wherein the detecting, from the camera system, the body part of the user associated with the task to be completed further comprises:
detecting one or more associated objects with the task, and
extracting a frame region around the body part and the one or more associated objects for each image frame of the camera system.
5. The method of claim 1 , further comprises training the machine learning model to detect sub-tasks by detecting the body part in the plurality of image frames, constructing image frame level features of respective image frames by flattening the body part with tree structures into a vector, maintain a buffer of image frame level features from a number of frames, evaluating the machine learning model on the image frame level features to predict sub-task labels and whether there is a change in sub-task, and clustering frame-level sub-tasks to determine segmentation and location of sub-tasks.
6. A non-transitory computer readable medium, storing instructions for executing a process, comprising:
analyzing a plurality of image frames, received from a camera system, using computer vision techniques to detect, in the plurality of image frames, a body part of a user associated with a task to be completed;
grouping sequential image frames of the plurality of image frames received from the camera system into a plurality of sub-tasks of the task to be completed based on an application of a machine learning model configured to associate each of the plurality of subtasks to the sequential image frames, said grouping comprises:
computing a dense optical flow over the plurality of image frames, computing the dense optical flow comprises producing x-component and a y-component images for each of the plurality of image frames;
associating each of the plurality of sub-tasks to sequential image frames through assigning sub-task labels to each of the sequential image frames based on using the machine learning model to classify each of the x-component and y-component of the sequential image frames using the machine learning model,
segmenting out each of the plurality of sub-tasks based on grouping the sub-task labels of the sequential image frames, and
grouping sequential ones of the segmented plurality of sub-tasks as the task to be completed; and
providing an evaluation of completion of the task based on detection of completion for the plurality of sub-tasks and order correctness of the sub-tasks, said evaluation comprises:
identifying an order of correctness of the sub-tasks from the grouping of sequential ones of the plurality of sub-tasks,
identifying completion for each of the plurality of sub-tasks through an application of a computer vision algorithm on corresponding ones of the labeled sequential image frames, and
computing a skill score for the task to be completed based on the identified order of correctness, the identified completion of the each of the plurality of sub-tasks, and a skill score for the each of the plurality of sub-tasks.
7. The non-transitory computer readable medium of claim 6 , wherein the detecting from the camera system, the body part of the user associated with the task to be completed comprises extracting a frame region around the body part for each image frame of the camera system.
8. The non-transitory computer readable medium of claim 6 , wherein the body part of the user comprises hands of the user, wherein the task to be completed is hand washing, wherein the plurality of sub-tasks comprises steps for conducting the hand washing.
9. The non-transitory computer readable medium of claim 6 , wherein the detecting, from the camera system, the body part of the user associated with the task to be completed further comprises:
detecting one or more associated objects with the task, and
extracting a frame region around the body part and the one or more associated objects for each image frame of the camera system.
10. A system, comprising:
a camera system; and
a processor, configured to:
analyze a plurality of image frames, received from a camera system, using computer vision techniques to detect, in the plurality of image frames, a body part of a user associated with a task to be completed;
group sequential image frames of the plurality of image frames received from the camera system into a plurality of sub-tasks of the task to be completed based on an application of a machine learning model configured to associate each of the plurality of subtasks to the sequential image frames, said grouping comprises:
compute a dense optical flow over the plurality of image frames, computing the dense optical flow comprises producing x-component and a y-component images for each of the plurality of image frames;
associate each of the plurality of sub-tasks to sequential image frames through assigning sub-task labels to each of the sequential image frames based on using the machine learning model to classify each of the x-component and y-component of the sequential image frames using the machine learning model,
segment out each of the plurality of sub-tasks based on grouping the sub-task labels of the sequential image frames, and
group sequential ones of the segmented plurality of sub-tasks as the task to be completed; and
provide an evaluation of completion of the task based on detection of completion for the plurality of sub-tasks and order correctness of the sub-tasks, said evaluation comprises:
identify an order of correctness of the sub-tasks from the grouping of sequential ones of the plurality of sub-tasks,
identify completion for each of the plurality of sub-tasks through an application of a computer vision algorithm on corresponding ones of the labeled sequential image frames, and
compute a skill score for the task to be completed based on the identified order of correctness, the identified completion of the each of the plurality of sub-tasks, and a skill score for the each of the plurality of sub-tasks.
11. The system of claim 10 , wherein the processor is configured to detect, from the camera system, the body part of the user associated with the task to be completed through extracting a frame region around the body part for each image frame of the camera system.
12. The system of claim 10 , wherein the body part of the user comprises hands of the user, wherein the task to be completed is hand washing, wherein the plurality of sub-tasks comprises steps for conducting the hand washing.
13. The system of claim 10 , wherein the processor is configured to detect, from the camera system, the body part of the user associated with the task to be completed by:
detecting one or more associated objects with the task, and
extracting a frame region around the body part and the one or more associated objects for each image frame of the camera system.