IP Library › Granted Patent US 12,704,838
Granted Patent B2
US 12,704,838 · App. 17/868,466 · Granted Aug 11, 2026

Gesture-controlled robotic feedback

Inventors: Avigal Denise Segal (Evergreen, CO); Andrew Jacob Petruska (Golden, CO); Anne Katherine Silverman (Golden, CO)
Assignee: Colorado School of Mines
G05D1/0016G05D1/0033
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,704,838
App. No.
17/868,466
Filed
Jul 19, 2022
Granted
Aug 11, 2026
Kind
B2
Art Unit
3668
USPC
701/2
Abstract

In one embodiment, a method of providing an intervention, includes: obtaining a motion sensing device configured for manipulation by a user; creating, via one or more processing elements, a communication link between the motion sensing device and a robot; and detecting, via the motion sensing device, a movement initiated by the user; creating, via the one or more processing elements, a signal within the sensing device; transmitting, via the one or more processing elements, the signal to the robot; and moving the robot based on the signal.

Claims (42)

1 . A device comprising:

a microcontroller;

an electronic transceiver;

a movement detector in communication with the microcontroller via the electronic transceiver; and

a robot, wherein the movement detector is configured to detect user motion and control the robot based on the motion, wherein the device is configured to be used for rehabilitation or diagnosis of a motor condition of the user, wherein the movement detector comprises at least one inertial measurement unit configured to assess the motor condition of the user.

2 . The device of claim 1 , wherein the movement detector comprises a wearable device.

3 . The device of claim 1 , wherein the movement detector is coupled to an intervention device.

4 . The device of claim 3 , wherein intervention device comprises a wobble board, the wobble board configured to receive the feet of the user thereupon and receive the motion based on a balance of the user.

5 . The device of claim 1 , wherein the robot comprises at least one of a tele-operated vehicle or an at least partially autonomous vehicle.

6 . The device of claim 5 , further comprising a course configured to receive the vehicle, wherein the control of the vehicle is configured to guide the vehicle through the course.

7 . A method of providing a rehabilitation or diagnosis for a user intervention, comprising:

obtaining a motion sensing device configured for manipulation by the user, wherein the motion sensing device is configured to assess a motor condition of the user;

creating, via one or more processing elements, a communication link between the motion sensing device and a robot; and

detecting, via the motion sensing device, a movement initiated by the user;

creating, via the one or more processing elements, a signal within the sensing device;

transmitting, via the one or more processing elements, the signal to the robot;

moving the robot based on the signal.

8 . The method of claim 7 , wherein the motion sensing device comprises a wearable device.

9 . The method of claim 7 , wherein the motion sensing device is coupled to an intervention device.

10 . The method of claim 9 , wherein the intervention device comprises a wobble board, the wobble board configured to receive the feet of the user thereupon and receive the motion based on a balance of the user.

11 . The method of claim 7 , wherein the robot comprises at least one of a tele-operated vehicle or an at least partially autonomous vehicle.

12 . The method of claim 7 , wherein the motion of the vehicle is configured to guide the vehicle through a course.

13 . The method of claim 7 , further comprising:

generating, via the motion sensing device, a signal operative to control the robot based on a user motion;

moving the robot in response to receiving the signal;

generates a visual feedback signal based on the motion of the robot; and

generating an updated signal, via the motion sensing device, based on an updated user motion and the visual feedback signal.

14 . A system;

a motion sensing device;

a robot; and

a central processing unit in electronic communication with at least one of the motion sensing device or the robot, wherein the system is configured to be used for rehabilitation or diagnosis of a motor condition of a user, wherein the movement detector comprises at least one inertial measurement unit configured to assess the motor condition of the user.

15 . The system of claim 14 , wherein:

the system comprises a closed loop control system;

the motion sensing device generates a signal operative to control the robot based on a user motion;

the robot moves in response to receiving the signal; and

the motion of the robot generates a visual feedback signal;

the motion sensing device generates an updated signal based on an updated user motion, the updated user motion based on the visual feedback signal.

16 . The system of claim 14 , wherein the motion sensing device comprises a wearable device.

17 . The system of claim 14 , wherein the motion sensing device is coupled to an intervention device.

18 . The system of claim 17 , wherein the intervention device comprises a wobble board, the wobble board configured to receive the feet of the user thereupon and receive the motion based on balance of the user.

19 . The system of claim 14 , wherein the robot comprises at least one of a tele-operated vehicle or an at least partially autonomous vehicle.

20 . The system of claim 14 , further comprising a course configured to receive the vehicle, wherein motion sensing device detects a user input and generates a signal configured to control of the vehicle to guide the vehicle through the course.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2026
From: SEGAL, AVA; PETRUSKA, ANDREW; SILVERMAN, ANNE
To: COLORADO SCHOOL OF MINES
Reel/Frame 073416/0310 →
Continuity (2)
Provisional Application 63223469 · Jul 19, 2021
Related Publication 20230037237A1 · Feb 2, 2023
References Cited (86)
US 10981279B2 · Hayashi · 2021 [cited by examiner]
US 11117047B2 · Lin · 2021 [cited by examiner]
US 12005981B2 · Doerksen · 2024 [cited by examiner]
US 20170160793A1 · Perlin · 2017 [cited by examiner]
US 20220291679A1 · Kimura · 2022 [cited by examiner]
US 20230037237A1 · Segal · 2023 [cited by examiner]
US 20230123518A1 · Sharma · 2023 [cited by examiner]
WO WO2020221311A1 · 2020 [cited by examiner]
Segal et al., iRebot: An Interactive Rehabilitation Robot with Gesture Control, 2020 IEEE (Year: 2020). [cited by examiner]
Fitzgerald et al., A Virtual Rehabilitation System for Wobble Board Balance Training with Children, 2008 IEEE (Year: 2008). [cited by examiner]
M. T. Wolf, C. Assad, M. T. Vernacchia, J. Fromm and H. L. Jethani, “Gesture-based robot control with variable autonomy from the JPL BioSleeve,” 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, … [cited by examiner]
Sathiyanarayanan, Mithileysh, et al. “Gesture controlled robot for military purpose.” International journal of technological research in Engineering 1 (2014): 1300-1303. (Year: 2014). [cited by examiner]
Balasubramanian, S.; Colombo, R.; Sterpi, I.; Sanguineti, V.; Burdet, E. Robotic assessment of upper limb motor function after stroke. Am. J. Phys. Med. Rehabil. 2012, 91, S255-S269. [cited by applicant]
Balasubramanian, S.; Melendez-Calderon, A.; Burdet, E. A robust and sensitive metric for quantifying movement smoothness. IEEE Trans. Biomed. Eng. 2012, 59, 2126-2136. [cited by applicant]
Barlett, N.D.; Gentile, D.A.; Barlett, C.P.; Eisenmann, J.C.; Walsh, D.A. Sleep as a mediator of screen time effects on US children's health outcomes: A prospective study. J. Child. Media 2012, 6, 37-50. [cited by applicant]
Birkenmeier, R.L.; Prager, E.M.; Lang, C.E. Translating animal doses of task-specific training to people with chronic stroke in 1-hour therapy sessions: A proof-of-concept study. Neurorehabil. Neural Rep. 2010, 24, 620-… [cited by applicant]
Borsci, S.; Federici, S.; Lauriola, M. On the dimensionality of the System Usability Scale: A test of alternative measurement models. Cog. Process. 2009, 10, 193-197. [cited by applicant]
Brewer, B.R.; McDowell, S.K.; Worthen-Chaudhari, L.C. Poststroke upper extremity rehabilitation: A review of robotic systems and clinical results. Top. Stroke Rehabil. 2007, 14, 22-44. [cited by applicant]
Broeks, J.G.; Lankhorst, G.J.; Rumping, K .; Prevo, A.J.H. The long-term outcome of arm function after stroke: Results of a follow-up study. Disabil. Rehabil. 1999, 21, 357-364. [cited by applicant]
Brooke, J. Sus: A Quick and Dirty usability scale. Usability Eval. Ind. 1996, 189, 4-7. [cited by applicant]
Brooke, J. Sus: A Retrospective. J. Usability Stud. 2013, 8, 29-40. [cited by applicant]
Brutsch, K.; Koenig, A.; Zimmerli, L.; Merillat-Koeneke, S.; Riener, R.; Jancke, L.; van Hedel, H.J.A.; Meyer-Heim, A. Virtual reality for enhancement of robot-assisted gait training in children with neurological gait d… [cited by applicant]
Charles, S.K.; Hogan, N. The curvature and variability of wrist and arm movements. Exp. Brain Res. 2010, 203, 63-73. [cited by applicant]
Chen, Y.; Duff, M.; Lehrer, N.; Sundaram, H.; He, J.; Wolf, S.L.; Rikakis, T. A computational framework for quantitative evaluation of movement during rehabilitation. AIP Conf. Proc. Am. Inst. Phys 2011, 1371, 317-326. [cited by applicant]
Colombo, R.; Pisano, F.; Micera, S.; Mazzone, A.; Delconte, C.; Carrozza, M.C.; Dario, P.; Minuco, G. Robotic techniques for upper limb evaluation and rehabilitation of stroke patients. IEEE Trans. Neural Sys. Rehabil. … [cited by applicant]
Duncan, P.W.; Horner, R.D.; Reker, D.M.; Samsa, G.P.; Hoenig, H.; Hamilton, B.; LaClair, B.J .; Dudley, T.K. Adherence to postacute rehabilitation guidelines is associated with functional recovery in stroke. Stroke 2002… [cited by applicant]
Feys, H.M.; De Weerdt, W.J.; Selz, B.E.; Cox Steck, G.A.; Spichiger, R.; Vereeck, L.E.; Putman, K.D.; Van Hoydonck, G.A. Effect of a therapeutic intervention for the hemiplegic upper limb in the acute phase after stroke… [cited by applicant]
Flash, T.; Hogan, N. The coordination of arm movements: An experimentally confirmed mathematical model. J. Neurosci. 1985, 5, 1688-1703. [cited by applicant]
Graves, L.E.; Ridgers, N.D.; Williams, K.; Stratton, G.; Atkinson, G.; Cable, N.T. The physiological cost and enjoyment of Wii Fit in adolescents, young adults, and older adults. J. Phys. Act. Health 2010, 7, 393-401. [cited by applicant]
Guadagnoli, M.A.; Lee, T.D. Challenge point: A framework for conceptualizing the effects of various practice conditions in motor learning. J. Mot. Behav. 2004, 36, 212-224. [cited by applicant]
Koch, Andreas, et al. “The neurophysiology and treatment of motion sickness.” Deutsches Ärzteblatt International 115.41 (2018): 687. [cited by applicant]
Lang, C.E.; MacDonald, J.R.; Reisman, D.S.; Boyd, L.; Kimberley, T.J.; Schindler-Ivens, S.M.; Hornby, T.G.; Ross, S. .; Scheets, P.L. Observation of amounts of movement practice provided during stroke rehabilitation. Ar… [cited by applicant]
Levin, M.F.; Weiss, P.L.; Keshner, E.A. Emergence of virtual reality as a tool for upper limb rehabilitation: Incorporation of motor control and motor learning principles. Phys. Ther. 2015, 95, 415-425. [cited by applicant]
Lewis, J.R.; Sauro, J. The factor structure of the System Usability Scale. In International Conference on Human Centered Design; Human Centered Design. HCD 2009. Lecture Notes in Computer Science; Kurosu, M.; Eds .; Spr… [cited by applicant]
Likert, R. A technique for the measurement of attitudes. Arch. Psychol. 1932, 140, 1-55. [cited by applicant]
Lissak, G. Adverse physiological and psychological effects of screen time on children and adolescents: Literature review and case study. Environ. Res. 2018, 164, 149-157. [cited by applicant]
Lohse, K.; Shirzad, N.; Verster, A.; Hodges, N.; Van der Loos, M. Video games and rehabilitation: Using design principles to enhance engagement in physical therapy. J. Neurol. Phys. Ther. 2013, 37, 166-175. [cited by applicant]
Manwell, Laurie A., et al. “Digital dementia in the internet generation: excessive screen time during brain development will increase the risk of Alzheimer's disease and related dementias in adulthood.” Journal of Integ… [cited by applicant]
Martinho, D., Carneiro, J., Corchado, J. M., & Marreiros, G. (2020). A systematic review of gamification techniques applied to elderly care. Artificial Intelligence Review, 53(7), 4863-4901. https://doi.org/10.1007/s104… [cited by applicant]
McLaughlin, A.; Gandy, M.; Allaire, J.; Whitlock, L. Putting fun into video games for older adults. Ergon. Des. 2012, 20, 13-22. [cited by applicant]
Nemec, D.; Janota, A.; Gregor, M.; Hrubos, M.; Pirnik, R. Control of the mobile robot by hand movement measured by inertial sensors. Electr. Eng. 2017, 99, 1161-1168. [cited by applicant]
Neophytou, Eliana, Laurie A. Manwell, and Roelof Eikelboom. “Effects of excessive screen time on neurodevelopment, learning, memory, mental health, and neurodegeneration: A scoping review.” International Journal of Ment… [cited by applicant]
Patel, K.; Bailenson, J.N.; Hack-Jung, S.; Diankov, R.; Bajcsy, R. The effects of fully immersive virtual reality on the learning of physical tasks. In Proceedings of the 9th Annual International Workshop on Presence, C… [cited by applicant]
Paulich, M.; Schepers, M.; Rudigkeit, N.; Bellusci, G. Xsens MTw Awinda: Miniature Wireless Inertial-Magnetic Motion Tracker for Highly Accurate 3D Kinematic Applications; Xsens: Enschede, The Netherlands, 2018; pp. 1-9. [cited by applicant]
Pomeroy, V.; Aglioti, S.M.; Mark, V.W.; McFarland, D.; Stinear, C.; Wolf, S.L.; Corbetta, M.; Fitzpatrick, S.M. Neurological principles and rehabilitation of action disorders: Rehabilitation interventions. Neurorehabil.… [cited by applicant]
Prahm, C.; Kayali, F.; Vujaklija, I.; Sturma, A.; Aszmann, O. Increasing motivation, effort and performance through game-based rehabilitation for upper limb myoelectric prosthesis control. In Proceedings of the 2017 Int… [cited by applicant]
Renewal Lodge.com. Available online: https://www.renewallodge.com/5-ways-quitting-drinking-affects-your-brain/ (accessed on Jun. 21, 2020). [cited by applicant]
Rodgers, J.L.; Nicewander, W.A. Thirteen ways to look at the correlation coefficient. Am. Stat. 1988, 42, 59-66. [cited by applicant]
Sathiyanarayanan, M.; Azharuddin, S.; Kumar, S.; Khan, G. Gesture controlled robot for military Purpose. Int. J. Technol. Res. Eng. 2014, 1, 1300-1303. [cited by applicant]
Schmeäl, Frank. “Neuronal mechanisms and the treatment of motion sickness.” Pharmacology 91.3-4 (2013): 229-241. [cited by applicant]
Segal, A.D. et al., Sensors, 2020. 20(4269): 1-18. [cited by applicant]
Segal, A.D.; Lesak, M.C.; Suttora, N.E.; Silverman, A.K.; Petruska, A.J. iRebot: An interactive rehabilitation robot with gesture control. In Proceedings of the IEEE International Conference Engineering in Medicine and … [cited by applicant]
Semprini, M.; Laffranchi, M.; Sanguineti, V.; Avanzino, L.; De Icco, R.; De Michieli, L.; Chiappalone, M. Technological approaches for neurorehabilitation: From robotic devices to brain stimulation and beyond. Front. Ne… [cited by applicant]
Shaw, Lindsay Alexander, et al. “Challenges in virtual reality exergame design.” (2015). [cited by applicant]
Sheppard, Amy L., and James S. Wolffsohn. “Digital eye strain: prevalence, measurement and amelioration.” BMJ open ophthalmology 3.1 (2018): e000146. [cited by applicant]
Sigrist, R.; Rauter, G.; Riener, R.; Wolf, P. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychon. Bull. Rev. 2013, 20, 21-53. [cited by applicant]
Sluijs, E.M.; Knibbe, J.J. Patient compliance with exercise: Different theoretical approaches to short-term and long-term compliance. Patient Educ. Couns. 1991, 17, 191-204. [cited by applicant]
Snoddy, G.S. Learning and stability: A psychophysiological analysis of a case of motor learning with clinical applications. J. App. Psyc. 1926, 10, 1. [cited by applicant]
Snodgrass, S.J.; Rivett, D.A.; Robertson, V.J.; Stojanovski, E. Real-time feedback improves accuracy of manually applied forces during cervical spine mobilisation. Man. Ther. 2010, 15, 19-25. [cited by applicant]
Standen, P.J.; Threapleton, K.; Connell, L.; Richardson, A.; Brown, D.J.; Battersby, S.; Sutton, C.J.; Platts, F. Patients' use of a home-based virtual reality system to provide rehabilitation of the upper limb followin… [cited by applicant]
Taylor, A.H.; May, S. Threat and coping appraisal as determinants of compliance with sports injury rehabilitation: An application of Protection Motivation Theory. J. Sport. Sci. 1996, 14, 471-482. [cited by applicant]
Taylor, R. Interpretation of the correlation coefficient: A basic review. J. Diagnos. Med. Sonogr. 1990, 6, 35-39. [cited by applicant]
Timmermans, A.A.; Seelen, H.A.; Willmann, R.D.; Kingma, H. Technology-assisted training of arm-hand skills in stroke: Concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology des… [cited by applicant]
Timmermans, A.A.A.; Seelen, H.A.M.; Geers, R.P.J.; Saini, P.K.; Winter, S.; te Vrugt, J.; Kingma, H. Sensor-based arm skill training in chronic stroke patients: Results on treatment outcome, patient motivation, stem usa… [cited by applicant]
Umejima, Keita, et al. “Paper notebooks vs. mobile devices: brain activation differences during memory retrieval.” Frontiers in Behavioral Neuroscience (2021): 34. [cited by applicant]
Warraich, Z.; Kleim, J.A. Neural plasticity: The biological substrate for neurorehabilitation. PM&R 2010, 2, S208-S219. [cited by applicant]
Wen, R.; Tay, W.L.; Nguyen, B.P.; Chng, C.B.; Chui, C.K. Hand gesture guided robot-assisted surgery based on a direct augmented reality interface. Comp. Meth. Prog. Biomed. 2014, 116, 68-80. [cited by applicant]
Wolf, M.T.; Assad, C.; Vernacchia, M.T.; Fromm, J.; Jethani, H.L. Gesture-based robot control with variable autonomy from the JPL BioSleeve. In Proceedings of the 2013 IEEE International Conference on Robotics and Autom… [cited by applicant]
Wu, C.Y.; Chen, C.I.; Tang, S.F.; Lin, K.C.; Huang, Y.Y. Kinematic and clinical analyses of upper-extremity movements after constraint-induced movement therapy in patients with stroke: A randomized controlled trial. Arc… [cited by applicant]
Wulf, G. Self-controlled practice enhances motor learning: Implications for physiotherapy. Physiotherapy 2007, 93, 96-101. [cited by applicant]
Yang, G.; Lv, H.; Chen, F.; Pang, Z.; Wang, J.; Yang, H.; Zhang, J. A novel gesture recognition system for intelligent interaction with a nursing-care assistant robot. Appl. Sci. 2018, 8, 2349. [cited by applicant]
Yue, Z.; Zhang, X.; Wang, J. Hand rehabilitation robotics on poststroke motor recovery. Behav. Neurol. 2017, 2017, 1-20. [cited by applicant]
Zhang, Li-Li, et al. “Motion sickness: current knowledge and recent advance.” CNS neuroscience & therapeutics 22.1 (2016): 15-24. [cited by applicant]
Zollo, L.; Rossini, L.; Bravi, M.; Magrone, G.; Sterzi, S.; Guglielmelli, E. Quantitative evaluation of upper-limb motor control in robot-aided rehabilitation. Med. Biolog. Eng. Comp. 2011, 49, 1131-1144. [cited by applicant]
Gamecho, Borja , “A Context-Aware Application to Increase Elderly Users Compliance with Physical Rehabilitation Exercises at Home via Animatronic Biofeedback”, Gamecho B, Silva H, Guerreiro J, Gardeazabal L, Abascal J. … [cited by applicant]
Geurts, Luc , “Digital games for physical therapy: fulfilling the need for calibration and adaptation”, Luc Geurts et al. Digital games for physical therapy: fulfilling the need for calibration and adaptation. TEI '11. … [cited by applicant]
Globe, Daniel J, “Using the Wii Fit as a tool for balance assessment and neurorehabilitation: the first half decade of “Wii-search.””, Goble DJ, Cone BL, Fling BW. Using the Wii Fit as a tool for balance assessment and … [cited by applicant]
Kappen, Dennis , “Older Adults' Physical Activity and Exergames: A Systematic Review”, Kappen D. L., “Older Adults' Physical Activity and Exergames: A Systematic Review” Int J Human-Computer Interact. 2019;35(2):140-167… [cited by applicant]
Kobeissi, Ahmad Hassan, “Development of a Hardware/Software System for Proprioception Exergaming”, Kobeissi AH, Lanza G, Berta R, Bellotti F, De Gloria A. Development of a hardware/software system for proprioception exe… [cited by applicant]
Kosse, Nienke , “Exergaming: Interactive balance training in healthy community-dwelling older adults”, Kosse NM, Caljouw SR, Vuijk P-J, Lamoth CJC. Exergaming: Interactive balance training in healthy community-dwelling … [cited by applicant]
Lamoth, Claudine , “Exergaming for elderly: effects of different types of game feedback on performance of a balance task”, Lamoth CJ, Alingh R, Caljouw SR. Exergaming for elderly: effects of different types of game feed… [cited by applicant]
Martinho, Diogo , “A systematic review of gamification techniques applied to elderly care”, Martinho, D., Carneiro, J., Corchado, J. M., & Marreiros, G. (2020). A systematic review of gamification techniques applied to … [cited by applicant]
Ogaya, Shinya , “Effects of balance training using wobble boards in the elderly”, Ogaya S, Ikezoe T, Soda N, Ichihashi N. Effects of balance training using wobble boards in the elderly. J Strength Cond Res. 2011;25(9):2… [cited by applicant]
Tahmosybayat, Robin , “Movements of older adults during exergaming interventions that are associated with the Systems Framework for Postural Control: A systematic review”, Tahmosybayat R, Baker K, Godfrey A, Caplan N, B… [cited by applicant]
Fitzgerald D, Trakarnratanakul N, Smyth B, Caulfield B. Effects of a wobble board-based therapeutic exergaming system for balance training on dynamic postural stability and intrinsic motivation levels. J Orthop Sports P… [cited by applicant]
Van Diest, Mike , “Exergaming for balance training of elderly: state of the art and future developments”, Van Diest M, Lamoth CJC, Stegenga J, Verkerke GJ, Postema K. Exergaming for balance training of elderly: state of… [cited by applicant]