Methods and systems for automatically creating statistically accurate ergonomics data
The systems and methods provide an action recognition and analytics tool for use in manufacturing, health care services, shipping, retailing and other similar contexts. Machine learning action recognition can be utilized to determine cycles, processes, actions, sequences, objects and or the like in one or more sensor streams. The sensor streams can include, but are not limited to, one or more video sensor frames, thermal sensor frames, infrared sensor frames, and or three-dimensional depth frames. The analytics tool can provide for analyzing ergonomic data from the one or more sensor streams.
1. A machine learning based ergonomics method comprising:
determining sensed activity information associated with a first actor and an activity space, wherein the sensed activity information includes at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters of a manufacturing operation and spatio-temporal data of the first actor, the spatio-temporal data comprising a location of the first actor and moments of work of the first actor, the moments of work comprising one or more of a weight, a torque, or a distance, the determining the sensed activity information comprising:
performing, with a frame feature extractor, a two-dimensional convolution operation on a video frame sensor stream to generate a two-dimensional array of feature vectors;
determining, with a region of interest detector, a dynamic region of interest in the video frame sensor stream, wherein the region of interest detector and the frame feature extractor share layers of a convolution neural network; and
processing an area of each video frame of the video frame sensor stream within the dynamic region of interest while discarding areas of the respective video frames outside the dynamic region of interest to determine the sensed activity information;
analyzing, by artificial intelligence, the determined sensed activity information for the first actor with respect to one or more ergonomic factors including work limit, work zone and hazard score; and
forwarding feedback based on the analyzing the determined activity information with respect to the one or more ergonomic factors, wherein:
the sensed activity information is received from sensors monitoring the activity space in real time, the sensors comprising a video sensor that produces the video frame sensor stream;
the determined activity information is analyzed in real time;
the feedback is forwarded in real time; and
the convolution neural network is applied to a plurality of sliding windows to determine the feedback with no computations repeated.
2. The method of claim 1 , wherein the feedback includes identification of ergonomically problematic activities.
3. The method of claim 2 , wherein the analyzing comprises:
comparing information associated with activity of the first actor within the activity space with identified representative actions; and
identifying a deviation between the activity of the first actor and a representative standard.
4. The method of claim 1 , further comprising:
determining sensed activity information associated with a second actor and the activity space, wherein the sensed activity information includes at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters of a manufacturing operation and spatio-temporal data of the second actor;
analyzing, by the artificial intelligence, the determined sensed activity information for the first actor and the second actor with respect to the one or more ergonomic factors including work limit, work zone and hazard score; and
forwarding feedback based on the analyzing the determined activity information with respect to the one or more ergonomic factors, wherein the feedback includes an identification of a selection between the first actor and the second actor.
5. The method of claim 4 , wherein the analyzing comprises:
determining if a deviation from a representative standard associated with a respective one of the first and second actors is within an acceptable threshold; and
identifying a respective one of a plurality of other actors as a potential acceptable candidate to be a replacement actor when the deviation associated with the respective one of the first and second actors is within an acceptable threshold.
6. One or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform a machine learning based ergonomics method comprising:
determining one or more data sets including one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of a manufacturing operation and spatio-temporal data of an actor, wherein the determining the one or more data sets comprises:
receiving a video frame sensor stream from a video sensor monitoring the manufacturing operation;
generating a two-dimensional array of feature vectors by performing a two-dimensional convolution operation on the video frame sensor stream with a frame extractor;
determining a dynamic region of interest (RoI) in the video frame sensor stream with an RoI detector, wherein a convolution neural network comprises convolution layers shared by the frame extractor and the RoI detector, wherein the dynamic RoI encloses an area of a video frame of the video frame sensor stream in which a specific action is occurring;
extracting a fixed-sized feature vector from the area within the dynamic RoI; and
analyzing actions within the fixed-sized feature vector of the video frame sensor stream while discarding areas of the video frame of the video frame sensor stream outside the fixed-sized feature vector to determine the one or more data sets;
accessing one or more ergonomic factors including a work limit, a work zone and a hazard score;
statistically analyzing by artificial intelligence in real time the one or more data sets based on the one or more ergonomic factors to determine an ergonomic data set;
adjusting at least one of the one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of the manufacturing operation in real time based on the ergonomic data set; and
forwarding feedback in real time based on the ergonomic data set, wherein:
the feedback is determined by applying the convolution neural network to a plurality of sliding windows such that no computations are repeated.
7. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the machine learning based ergonomics method according to claim 6 , further comprising:
storing the ergonomic data set indexed to corresponding portions of one or more sensor streams, the one or more sensor streams comprising the video stream; and
storing the corresponding portions of the one or more sensor streams.
8. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the machine learning based ergonomics method according to claim 7 , wherein the ergonomic data set and the corresponding portions of the one or more sensor streams are blockchained.
9. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the machine learning based ergonomics method according to claim 6 , further comprising:
selecting one of a plurality of actors based on the ergonomic data set.
10. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the machine learning based ergonomics method according to claim 6 , wherein the one or more indicators of at least one of the one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters in the one or more data sets include one or more locations of one or more portions of the actor in a workspace.
11. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the machine learning based ergonomics method according to claim 10 , wherein:
the hazard scores of the one or more ergonomic factors include hazard scores for a plurality of zones of the workspace, wherein at least two zones of the workspace have different hazard scores.
12. A system comprising:
one or more data storage units;
a video sensor configured to produce a video frame sensor stream; and
a computing circuit configured to:
determine one or more data sets in the one or more data storage units including one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects and one or more parameters of a manufacturing operation and spatio-temporal data of a first action, wherein the one or more data sets are determined by:
performing a two-dimensional convolution operation on the video frame sensor stream to generate a two-dimensional array of feature vectors through a frame feature extractor;
combining neighboring ones of the feature vectors to determine a dynamic region of interest (RoI) in the video frame sensor stream through an RoI detector, wherein the RoI detector and the frame feature extractor share layers of a convolution neural network; and
extracting a fixed-sized feature vector from an area of a video frame of the video frame sensor stream within the dynamic RoI and discarding the remaining feature vectors of the video frame, wherein the convolution neural network analyzes actions in the video frame within the dynamic RoI to determine the one or more data sets;
access one or more ergonomic factors in the one or more data storage units including a work limit, a work zone and a hazard score;
statistically analyze in real time by one or more processing units the one or more data sets based on the one or more ergonomic factors to determine an ergonomic data set;
store the ergonomic data set indexed to corresponding portions of the one or more data sets in the one or more data structures on the one or more data storage units; and
forward feedback in real time based on the ergonomic data set, wherein:
the convolution neural network determines the feedback in a plurality of sliding windows without repeating computations.
13. The system of claim 12 , wherein the computing circuit is further configured to:
blockchain the ergonomic data set and the corresponding portions of the one or more data sets.
14. The system of claim 12 , wherein:
the one or more data sets include one or more data sets for a plurality of actors; and
the computing circuit is further configured to:
select one of a plurality of actors based on the ergonomic data set.
15. The system of claim 12 , wherein the ergonomic data set includes one or more of a reach study, a motion study, a repetitive motion study, and a dynamics study.