IP Library › Granted Patent US 12,232,547
Granted Patent B2
US 12,232,547 · App. 17/440,250 · Granted Feb 25, 2025

Tracking and alert method and system for worker productivity and safety

Inventor: Yong Cho (Atlanta, GA)
Assignee: Georgia Tech Research Corporation
A41D1/002B60Q1/525F16P3/147G01S11/06G06N3/044G06Q10/06398G08B7/06G08B21/0446
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,232,547
App. No.
17/440,250
Granted
Feb 25, 2025
Kind
B2
Abstract

An exemplary method and system is disclosed that facilitates the monitoring of worker's productivity and safety, for example, at a construction site and other like labor-intensive occupation and settings. The exemplary method and system provides wearable-based motion sensing and/or proximity sensing between worker and construction equipment. A study discussed herein suggests that sensing at two or more torso locations can provide 95% accuracy in classifying worker's action or motion at a construction site. In some embodiments, as discussed herein, the acquired sensed data are transmitted, over a mesh network, e.g., established between the wearable devices, to a cloud infrastructure to facilitate the real-time monitoring of action and actions that such sites.

Claims (41)

1. A productivity and safety tracking system comprising:

a plurality of portable personal tracking apparatus attachable to a safety vest or garment, including a first portable personal tracking apparatus and a second portable personal tracking apparatus, wherein the first portable personal tracking apparatus is attachable to the safety vest or garment at a first torso location of the safety vest or garment, wherein the second portable personal tracking apparatus is attachable to the safety vest or garment at a second torso location of the safety vest or garment, and

wherein each of the first and second portable personal tracking apparatus comprises:

a respective inertial measurement sensor unit configured to measure a measurand selected from the group consisting of acceleration, angular velocity, and magnetic field;

a respective radio transceiver;

a controller unit operatively coupled to the respective radio transceiver and respective inertial measurement sensor unit;

wherein the first portable personal tracking apparatus is configured, by computer readable instructions, to generate a first inertial measurement data set associated with motion of the first torso location, wherein the second portable personal tracking apparatus is configured, by computer readable instructions, to generate a second inertial measurement data set associated with motion of the second torso location, and wherein the first and second inertial measurement data sets are concatenated for each of a plurality of nodes to form a respective feature set, wherein feature sets for each of the plurality of nodes are concatenated to form an input vector for classifier training or classification; and

wherein the input vector is subsequently used, in a motion recognition classification operation, to identify a sequenced motion from a set of candidate sequenced motions of a person.

2. The productivity and safety tracking system of claim 1 , wherein the motion recognition classification operation is performed by i) a first machine learning classification operation trained from first data sets associated with motion and movement of a set of persons to identify one or more movement instances and ii) a second machine learning classification operation trained from second data sets associated with motion and movement of a set of persons to identify the candidate sequenced motions using the identified one or more movement instances.

3. The productivity and safety tracking system of claim 2 , wherein the first machine learning classification operation is selected from the group consisting of logistic regression, k-nearest neighbor, multilayer perceptron, random forest, and support vector machine.

4. The productivity and safety tracking system of claim 2 , wherein the second machine learning classification operation is performed by an artificial recurrent neural network or a multi-stacked Long Short-Term Memory (LSTM) network model.

5. The productivity and safety tracking system of claim 2 , wherein the second machine learning classification operation is performed by a model configured to learn sequential information having temporal relationships on a long-time scale.

6. The productivity and safety tracking system of claim 2 , wherein the first and second data sets associated with motion and movement includes, at least, measured data associated with a person standing, bending-up, bending, bending-down, squatting-up, squatting, walking, twisting, working overhead, and kneeling.

7. The productivity and safety tracking system of claim 1 , wherein the system further comprises a third portable personal tracking apparatus, wherein the third portable personal tracking apparatus is attachable to an extremity, wherein the third portable personal tracking apparatus is configured, by computer readable instructions, to generate a third inertial measurement data set associated with motion of the extremity, and wherein the first, second, and third inertial measurement data sets are subsequently used, in the motion recognition classification operation, to identify the sequenced motion from the set of candidate sequenced motions of a person.

8. The productivity and safety tracking system of claim 1 , wherein the respective inertial measurement sensor unit of the first portable personal tracking apparatus includes, at least, a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis digital compass.

9. The productivity and safety tracking system of claim 1 , wherein the controller unit of the first portable personal tracking apparatus and is operatively coupled to the respective radio transceiver to determine a signal strength value for a transmitted signal sent from a low-power short-range beacon,

the first portable personal tracking apparatus further comprising a low-power communication device to establish a plurality of links with one or more portable personal tracking apparatuses of other productivity and safety tracking system to form a mesh network,

wherein the controller unit of the first portable personal tracking apparatus is configured, by computer readable instructions, to transmit, at least, the first inertial measurement data set in a plurality of datagrams, through the mesh network, to a gateway networking device, and

wherein the gateway networking device is configured to transmit received proximity event data messages to one or more remote computing devices.

10. The productivity and safety tracking system of claim 1 , wherein at least one of the remote computing devices comprises a cloud server, a remote server, or a local server, wherein the at least one remote computing device is configured to store transmitted proximity event data to be subsequently presented, through curation operation performed by the one or more remote computing devices or another computer device, at a monitoring application executing at a monitoring terminal.

11. The productivity and safety tracking system of claim 1 , wherein at least one of the plurality of portable personal tracking apparatus is configured to generate an audible and/or vibratory alert based on sensed proximity to safety-associated beacon.

12. The productivity and safety tracking system of claim 1 , wherein the plurality of portable personal tracking apparatus includes a personal or equipment protection apparatus comprising:

a radio transceiver;

an ultra-low power multi-protocol system-on-module (SOM) unit operatively coupled to the radio transceiver to determine a signal strength value for a transmitted signal sent from a low-power short-range beacon; and

one or more alert devices each operatively coupled to the ultra-low power multi-protocol system-on-module (SOM) unit, wherein the one or more alert devices is configured, by computer readable instructions, to generate warning sound, a warning visual output, or a vibrational output based on the determined signal strength value, wherein the determined signal strength value is indicative of the personal or equipment protection apparatus being within a predetermined proximity to the low-power short-range beacon.

13. A method of monitoring productivity and safety comprising:

retrieving, through a network, at a monitoring computing device, two or more inertial measurement data sets having been acquired from a portable personal tracking apparatus attached to a safety vest or garment including at least a first inertial measurement data set and a second inertial measurement data set, wherein the two or more inertial measurement data sets includes inertial measurement data acquired at a first torso location of the safety vest or garment and inertial measurement data acquired at a second torso location of the safety vest or garment, and wherein the first and second inertial measurement data sets are concatenated for each of a plurality of nodes to form a respective feature set, wherein feature sets for each of the plurality of nodes are concatenated to form an input vector for classifier training or classification;

classifying, at the monitoring computing device, sequenced motion of a person wearing the safety vest or garment using the retrieved two or more inertial measurement data sets, wherein the classified sequenced motion is selected from the group of a person standing, bending-up, bending, bending-down, squatting-up, squatting, walking, twisting, working overhead, and kneeling; and

storing, at the monitoring computing device, the classified sequenced motion, wherein the classified sequenced motion is subsequently curated to at a monitoring application executing at a monitoring terminal for the monitoring of productivity or safety of the person.

14. The method of monitoring productivity and safety of claim 13 further comprising:

presenting, at the monitoring terminal, via a web-based or local GUI interface, frequency or duration of a given sequenced motion for a given work shift, day, week, or month.

15. The method of monitoring productivity and safety of claim 14 further comprising:

presenting, at the monitoring terminal, via the web-based or local GUI interface, task associated sequenced motions for the given work shift, day, week, or month, wherein task associated sequenced motions comprises two or more defined sequenced motions associated with a given productivity task.

16. The method of monitoring productivity and safety of claim 14 further comprising:

presenting, at the monitoring terminal, via the web-based or local GUI interface, frequency or duration of a safety event associated with a sensed proximity to safety-associated beacon for the given work shift, day, week, or month.

17. The method of monitoring productivity and safety of claim 13 further comprising:

acquiring, at the portable personal tracking apparatus, the one or more inertial measurement data sets, wherein the portable personal tracking apparatus an inertial measurement sensor unit that includes, at least, a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis digital compass.

18. The method of monitoring productivity and safety of claim 13 further comprising:

transmitting, from the portable personal tracking apparatus, to the monitoring computing device, the one or more inertial measurement data sets through a mesh network, as the network, established across a plurality of portable personal tracking apparatuses.

19. The method of monitoring productivity and safety of claim 13 further comprising:

generating, at the monitoring terminal, an alert for a sensed motion associated with a person lying down.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 8, 2024
From: CHO, YONG
To: GEORGIA TECH RESEARCH CORPORATION
Reel/Frame 068834/0918 →
Continuity (3)
Provisional Application 62820047 · Mar 18, 2019
Provisional Application 62820040 · Mar 18, 2019
Related Publication 20220172594A1 · Jun 2, 2022
References Cited (63)
US 20040031083A1 · Copp · 2004 [cited by applicant]
US 20100289662A1 · Dasilva et al. · 2010 [cited by applicant]
US 20110227748A1 · Schaible et al. · 2011 [cited by applicant]
US 20120293382A1 · Matthews et al. · 2012 [cited by applicant]
US 20140070957A1 · Longinotti-Buitoni et al. · 2014 [cited by applicant]
US 20140228649A1 · Rayner · 2014 [cited by examiner]
US 20150091716A1 · Hathaway et al. · 2015 [cited by applicant]
US 20160091965A1 · Wang · 2016 [cited by examiner]
US 20160301581A1 · Carter · 2016 [cited by examiner]
US 20190339766A1 · Erivantcev · 2019 [cited by examiner]
US 20200174517A1 · Martinez · 2020 [cited by examiner]
US 20200258365A1 · Ten Kate · 2020 [cited by examiner]
DIGI, Emerging Requirement for Next generation SOMs and SBC (Year: 2016). [cited by examiner]
International Search Report and Written Opinion dated Jul. 28, 2020, from International Application No. PCT/US2020/023270, 12 pages. [cited by applicant]
Goodrum, P. M., McLaren, M. A., and Durfee, A. (2006). “The application of active radio frequency identification technology for tool tracking on construction job sites.” Automation in Construction, 15(3), 292-302. [cited by applicant]
Lazaro, A., Girbau, D., and Salinas, D. (2009). “Radio Link Budgets for UHF RFID on Multipath Environments.” IEEE Transactions on Antennas and Propagation, 57(4), 1241-1251. [cited by applicant]
Park, J., Marks, E., Cho, Y., and Suryanto, W. (2015b). “Performance Test of Wireless Technologies for Personnel and Equipment Proximity Sensing in Work Zones.” ASCE Journal of Construction Engineering and Management. [cited by applicant]
Ruff, T. (2007). Recommendations for evaluating and implementing proximity warning system on surface mining equipment. [cited by applicant]
Marks, E. (2014). “Active Safety Leading Indicators for Human-Equipment Interaction on Construction Sites.” Doctoral Dissertation, Georgia Institute of Technology. [cited by applicant]
CPWR—The Center for Construction Research and Training, The Construction Chart Book: The U.S. Construction Industry and Its Workers, 6th ed., 2018. (Jul. 17, 2019). [cited by applicant]
The National Institute for Occupational Safety and Health (NIOSH) Ergonomics and Musculoskeletal Disorders, Workplace Safety & Health (Jul. 18, 2019). [cited by applicant]
McKinsey & Company, The Construction Productivity Imperative, McKinsey & Company, 2015 (Jan. 22, 2018). [cited by applicant]
P.R. Ghate, A.B. More, P.R. Minde, “Importance of measurement of labour productivity in construction” (Jul. 2019). [cited by applicant]
McKinsey Global Institute “Reinventing construction: a route to higher productivity” (Feb. 2020). [cited by applicant]
L. Sveikauskas, S. Rowe, J. Mildenberger, J. Price, A. Young, “Measuring productivity growth in construction,” Monthly Labor Review (2018). [cited by applicant]
L. Pei, R. Guinness, J. Kaistinen, “Cognitive phone for sensing human behavior,” IGI Global, Hershey, PA, USA, 2015, pp. 1138-1150. [cited by applicant]
S. Han, S.M. Asce, S. Lee, A.M. Asce, F. Peña-Mora, M. Asce, “Vision-based detection of unsafe actions of a construction worker: case study of ladder climbing,” J. Comput. Civ. Eng. 27 (2013) 635-644. [cited by applicant]
J. Yang, Z. Shi, Z. Wu, “Vision-based action recognition of construction workers using dense trajectories,” Adv. Eng. Inform. 30 (2016) 327-336. [cited by applicant]
L. Ding, W. Fang, H. Luo, P.E.D. Love, B. Zhong, X. Ouyang, “A deep hybrid learning model to detect unsafe behavior: integrating convolution neural networks and long short-term memory,” Autom. Constr. 86 (2018) 118-124. [cited by applicant]
K. Kim, H. Kim, H. Kim, “Image-based construction hazard avoidance system using augmented reality in wearable device,” Autom. Constr. 83 (2017) 390-403. [cited by applicant]
H. Guo, Y. Yu, Q. Ding, M. Skitmore, “Image-and-skeleton-based parameterized approach to real-time identification of construction workers' unsafe behaviors,” J. Constr. Eng. Manag. 144 (2018). [cited by applicant]
H. Zhang, X. Yan, H. Li, “Ergonomic posture recognition using 3D view-invariant features from single ordinary camera,” Autom. Constr. 94 (2018) 1-10. [cited by applicant]
J.A. Hess, L. Kincl, T. Amasay, P. Wolfe, “Ergonomic evaluation of masons laying concrete masonry units and autoclaved aerated concrete,” Appl. Ergon. 41 (2010) 477-483. [cited by applicant]
H. Luo, C. Xiong, W. Fang, P.E.D. Love, B. Zhang, X. Ouyang, “Convolutional neural networks: computer vision-based workforce activity assessment in construction,” Autom. Constr. 94 (2018) 282-289. [cited by applicant]
V. Escorcia, M.A. Dávila, M. Golparvar-Fard, J.C. Niebles, “Automated vision-based recognition of construction worker actions for building interior construction operations using RGBD cameras,” Construction Research Cong… [cited by applicant]
Y. Yu, H. Guo, Q. Ding, H. Li, M. Skitmore, “An experimental study of real-time identification of construction workers' unsafe behaviors,” Autom. Constr. 82 (2017) 193-206. [cited by applicant]
S. Han, S. Lee, “A vision-based motion capture and recognition framework for behavior-based safety management,” Autom. Constr. 35 (2013) 131-141. [cited by applicant]
A. Golabchi, S. Han, S. AbouRizk, “A simulation and visualization-based framework of labor efficiency and safety analysis for prevention through design and planning,” Autom. Constr. 96 (2018) 310-323. [cited by applicant]
J. Chen, J. Qiu, C. Ahn, “Construction worker's awkward posture recognition through supervised motion tensor decomposition,” Autom. Constr. 77 (2017) 67-81. [cited by applicant]
R. Akhavian, A.H. Behzadan, “Smartphone-based construction workers' activity recognition and classification,” Autom. Constr. 71 (2016) 198-209. [cited by applicant]
N.D. Nath, A.H. Behzadan, “Construction productivity and ergonomic assessment using mobile sensors and machine learning,” Computing in Civil Engineering, 2017, American Society of Civil Engineers, Reston, VA, 2017, pp. … [cited by applicant]
R. Akhavian, A.H. Behzadan, “Productivity analysis of construction worker activities using smartphone sensors,” Proceedings of 16th International Conference on Computing in Civil and Building Engineering, 2016, pp. 1067… [cited by applicant]
M.F. Antwi-Afari, H. Li, J. Seo, A.Y.L. Wong, “Automated detection and classification of construction workers' loss of balance events using wearable insole pressure sensors,” Autom. Constr. 96 (2018) 189-199. [cited by applicant]
S. Xing, H. Tong, P. Ji, “Activity recognition with smartphone sensors,” Tsinghua Sci. Technol. 19 (2014) 235-249. [cited by applicant]
J. Ryu, J. Seo, M. Liu, S. Lee, C.T. Haas, “Action recognition using a wristband-type activity tracker: case study of masonry work,” Construction Research Congress, 2016, 2016, pp. 790-799. [cited by applicant]
K. Yang, C.R. Ahn, “Inferring workplace safety hazards from the spatial patterns of workers' wearable data,” Adv. Eng. Inform. 41 (2019). [cited by applicant]
J. Ryu, J. Seo, H. Jebelli, S. Lee, “Automated action recognition using an accelerometer-embedded wristband-type activity tracker,” J. Constr. Eng. Manag. 145 (2019) 1-14. [cited by applicant]
K. Yang, C.R. Ahn, M.C. Vuran, S.S. Aria, “Semi-supervised near-miss fall detection for ironworkers with a wearable inertial measurement unit,” Autom. Constr. 68 (2016) 194-202. [cited by applicant]
A. Alwasel, A. Sabet, M. Nahangi, C.T. Haas, E. Abdel-Rahman, “Identifying poses of safe and productive masons using machine learning,” Autom. Constr. 84 (2017) 345-355. [cited by applicant]
J. Zhao, E. Obonyo, “Convolutional long short-term memory model for recognizing postures from wearable sensor,” CEUR Workshop Proceedings, 2019. [cited by applicant]
E. Valero, A. Sivanathan, F. Bosché, M. Abdel-Wahab, “Analysis of construction trade worker body motions using a wearable and wireless motion sensor network,” Autom. Constr. 83 (2017) 48-55. [cited by applicant]
K. Yang, C.R. Ahn, H. Kim, “Validating ambulatory gait assessment technique for hazard sensing in construction environments,” Autom. Constr. 98 (2019) 302-309. [cited by applicant]
H. Jebelli, C.R. Ahn, T.L. Stentz, “Fall risk analysis of construction workers using inertial measurement units: validating the usefulness of the postural stability metrics in construction,” Saf. Sci. 84 (2016) 161-170. [cited by applicant]
H. Jebelli, S.M. Asce, Changbum, R. Ahn, A.M. Asce, T.L. Stentz, “Comprehensive fall-risk assessment of construction workers using inertial measurement units: validation of the gait-stability metric to assess the fall r… [cited by applicant]
X. Yan, H. Li, A.R. Li, H. Zhang, “Wearable IMU-based real-time motion warning system for construction workers' musculoskeletal disorders prevention,” Autom. Constr. 74 (2017) 2-11. [cited by applicant]
A. Singh, N. Thakur, A. Sharma, “A review of supervised machine learning algorithms,” Proceedings of the 10th INDIACom, 2016 3rd International Conference on Computing for Sustainable Global Development, INDIACom 2016, 2… [cited by applicant]
S. Dreiseitl, L. Ohno-Machado, “Logistic regression and artificial neural network classification models: a methodology review,” J. Biomed. Inform. 35 (2002) 352-359. [cited by applicant]
S.-R. Ke, H. Thuc, Y.-J. Lee, J.-N. Hwang, J.-H. Yoo, K.-H. Choi, “A review on video-based human activity recognition,” Computers 2 (2013) 88-131. [cited by applicant]
F. Attal, S. Mohammed, M. Dedabrishvili, F. Chamroukhi, L. Oukhellou, Y. Amirat, “Physical human activity recognition using wearable sensors,” Sensors 15 (2015) 31314-31338. [cited by applicant]
F. Ordóñez, D. Roggen, “Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition,” Sensors 16 (2016) 115. [cited by applicant]
Y. Zhao, R. Yang, G. Chevalier, M. Gong, “Deep residual Bidir-LSTM for human activity recognition using wearable sensors,” Math. Probl. Eng. 2018 (2018) 13. [cited by applicant]
D.P. Kingma, J. Lei Ba, “ADAM: a method for stochastic optimization,” The 3rd International Conference for Learning Representations, San Diego, 2015. [cited by applicant]
K. Boyd, K.H. Eng, C.D. Page, “Area Under the Precision-recall Curve: Point Estimates and Confidence Intervals,” Springer, Berlin, Heidelberg, 2013. [cited by applicant]