Vision-based hand grip recognition method and system for industrial ergonomics risk identification
A system, comprising: a computing device configured to obtain video signals of a worker performing a hand-related job at a workplace; and a computing server system configured to receive and process the video signals to identify hand grips and wrist bending involved in the job, determine a hand grip type for each identified hand grip, obtain force information relating to each identified hand grip, determine neutral or hazardous wrist bending based at least upon the wrist bending and the hand grip force information, calculate a percent maximum strength for each identified hand grip, calculate frequencies and durations of each identified hand grip and wrist bending, and determine ergonomic risks of the hand-related job accordingly.
1. A system comprising:
a computing device comprising:
a non-transitory computer-readable storage medium configured to store an application program; and
a processor coupled to the non-transitory computer-readable storage medium and configured to control a plurality of modules to execute instructions of the application program to obtain video signals of a worker performing a work task at a workplace; and
a computing server system configured to:
receive the video signals,
obtain one or more image frames from the video signals,
incorporate a deep learning model to identify joint positions and postures of a body part involved in the work task based at least upon the one or more image frames, wherein the deep learning model is trained and validated by:
recording a number of body part-related activities to obtain images capturing different joint positions and postures of the body part,
determining 3D coordinates of joints in each image based at least upon a tracking confidence set for at least one detected body part in sequential images, and
normalizing the 3D coordinates of joints to calculate confidence values corresponding to a number of postures of the body part,
calculate a frequency and a duration of each identified posture of the body part,
determine ergonomic risks of the work task based at least upon the frequency and the duration of each identified posture of the body part, and
present, via the application program of the computing device, assessment results of the ergonomic risks to a user.
2. The system of claim 1 , wherein the application program of the computing device is configured to directly record the video signals of the worker performing the work task.
3. The system of claim 1 , wherein the application program of the computing device is configured to obtain the video signals of the worker performing the work task captured by another computing or imaging device.
4. The system of claim 1 , wherein the computing server system is further configured to determine a level of the ergonomic risks associated with the body part, wherein the level of the ergonomic risks associated with the body part is one of a low, medium or high level.
5. The system of claim 1 , wherein the computing server system is configured to use the deep learning model to identify the joint positions of the body part by determining X, Y, and Z coordinates of each joint position of the body part in the one or more image frames.
6. The system of claim 5 , wherein the X, Y, and Z coordinates of each joint position are overlaid on the body part to visualize the ergonomic risks.
7. The system of claim 1 , wherein the computing server system is further configured to provide ergonomic risk control recommendations to mitigate the ergonomic risks.
8. A computer-implemented method comprising:
obtaining, by a processor of a computing device deployed within a communication network, video signals of a worker performing a work task at a workplace;
receiving, by a computing server system deployed within the communication network, the video signals;
obtaining, by the computing server system, one or more image frames from the video signals;
incorporating, by the computing server system, a deep learning model to identify joint positions and postures of a body part involved in the work task based at least upon the one or more image frames, wherein the deep learning model is trained and validated by:
recording a number of body part-related activities to obtain images capturing different joint positions and postures of the body part,
determining 3D coordinates of joints in each image based at least upon a tracking confidence set for at least one detected body part in sequential images, and
normalizing the 3D coordinates of joints to calculate confidence values corresponding to a number of postures of the body part;
calculating a frequency and a duration of each identified posture of the body part;
determining ergonomic risks of the work task based at least upon the frequency and the duration of each identified posture of the body part; and
presenting, via the computing device, assessment results of the ergonomic risks to a user.
9. The computer-implemented method of claim 8 , wherein obtaining, by the computing device, the video signals comprises directly recording, by an application program of the computing device, the video signals of the worker performing the work task.
10. The computer-implemented method of claim 8 , wherein the video signals are obtained from another computing or imaging device.
11. The computer-implemented method of claim 8 , further comprising determining a level of the ergonomic risks associated with the body part, wherein the level of the ergonomic risks associated with the body part is one of a low, medium or high level.
12. The computer-implemented method of claim 8 , wherein incorporating, by the computing server system, the deep learning model to identify the joint positions of the body part comprises determining X, Y, and Z coordinates of each joint position of the body part in the one or more image frames.
13. The computer-implemented method of claim 12 , wherein the X, Y, and Z coordinates of each joint position are overlaid on the body part to visualize the ergonomic risks.
14. The computer-implemented method of claim 8 , further comprising providing, by the computing server system, ergonomic risk control recommendations to mitigate the ergonomic risks.
15. A non-transitory computer readable medium storing computer executable instructions for a system deployed within a communication network, the instructions being configured for:
obtaining, by a processor of a computing device deployed within a communication network, video signals of a worker performing a work task at a workplace;
receiving, by a computing server system deployed within the communication network, the video signals;
obtaining, by the computing server system, one or more image frames from the video signals;
incorporating, by the computing server system, a deep learning model to identify joint positions and postures of a body part involved in the work task based at least upon the one or more image frames, wherein the deep learning model is trained and validated by:
recording a number of body part-related activities to obtain images capturing different joint positions and postures of the body part,
determining 3D coordinates of joints in each image based at least upon a tracking confidence set for at least one detected body part in sequential images, and
normalizing the 3D coordinates of joints to calculate confidence values corresponding to a number of postures of the body part;
calculating a frequency and a duration of each identified posture of the body part;
determining ergonomic risks of the work task based at least upon the frequency and the duration of each identified posture of the body part; and
presenting, via the computing device, assessment results of the ergonomic risks to a user.
16. The non-transitory computer readable medium of claim 15 , wherein the instructions for obtaining, by the computing device, the video signals comprise instructions for directly recording, by an application program of the computing device, the video signals of the worker performing the work task.
17. The non-transitory computer readable medium of claim 15 , wherein the video signals are obtained from another computing or imaging device.
18. The non-transitory computer readable medium of claim 15 , further comprising determining a level of the ergonomic risks associated with the body part, wherein the level of the ergonomic risks associated with the body part is one of a low, medium or high level.
19. The non-transitory computer readable medium of claim 15 , wherein the instructions for incorporating, by the computing server system, the deep learning model to identify the joint positions of the body part comprise instructions for determining X, Y, and Z coordinates of each joint position of the body part in the one or more image frames, wherein the X, Y, and Z coordinates of each joint position are overlaid on the body part to visualize the ergonomic risks.
20. The non-transitory computer readable medium of claim 15 , further comprising instructions for providing, by the computing server system, ergonomic risk control recommendations to mitigate the ergonomic risks.