Gesture control via eye tracking, head tracking, facial expressions and other user actions
Methods of interpreting user actions for controlling electronic devices, as well as apparatuses and systems implementing the methods. User actions can involve eyes, head, face, fingers, hands, arms, other body parts, as well as facial, verbal and/or mental actions. Hand and/or voice free control of devices. Methods for large as well as fine motion and placement of objects. Use of actions before and during other actions to confirm the intent of the other actions. Objects can be moved or warped based on combinations of actions. Measurement of eye gaze, and iterations and helper signals for improved accuracy of control. Triggers to start and end recognition of user actions and generation of device commands.
1. A computer implemented method of controlling an electronic device by a user, the computer implemented method comprising:
monitoring motions of the user's head;
monitoring user's eye gaze;
when the motions of the user's head reach or cross a first head motion threshold to go outside a head motion tolerance zone, generating one or more first commands to move an object of interest (OOI) in accordance to the user's eye gaze; and
when the motions of the user's head stay within the head motion tolerance zone, not generating the one or more first commands to move the OOI in accordance to the user's eye gaze;
wherein the OOI is affected by the electronic device.
2. The computer implemented method of claim 1 , wherein monitoring motions of the user's head comprises monitoring the speed of the user's head and wherein the first head motion threshold is a speed threshold.
3. The computer implemented method of claim 2 , further comprising:
after generating the one or more first commands, generating one or more second commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a second head motion threshold.
4. The computer implemented method of claim 1 , further comprising:
after generating the one or more first commands, generating one or more third commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a third head motion threshold.
5. The computer implemented method of claim 1 , further comprising:
after generating the one or more first commands, generating one or more fourth commands to move the OOI;
wherein the one or more fourth commands are generated in accordance to the motions of the user's head when the user's eye gaze moves by less than a first eye gaze motion threshold; and
wherein the one or more fourth commands are generated in accordance to the user's eye gaze when the user's eye gaze moves by greater than or equal to the first eye gaze motion threshold.
6. The computer implemented method of claim 5 , further comprising:
stop generating the one or more fourth commands when the motions of the user's head fall under a fourth head motion threshold.
7. The computer implemented method of claim 1 , further comprising:
after generating the one or more first commands, generating one or more fifth commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a fifth head motion threshold or until the user's eye gaze moves by greater than or equal to a second eye gaze motion threshold.
8. A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, the one or more programs causing performance of a method comprising:
monitoring motions of the user's head;
monitoring user's eye gaze;
when the motions of the user's head reach or cross a first head motion threshold to go outside a head motion tolerance zone, generating one or more first commands to move an object of interest (OOI) in accordance to the user's eye gaze; and
when the motions of the user's head stay within the head motion tolerance zone, not generating the one or more first commands to move the OOI in accordance to the user's eye gaze;
wherein the OOI is affected by the electronic device.
9. The computer readable medium of claim 8 , wherein monitoring motions of the user's head comprises monitoring speed of the user's head and wherein the first head motion threshold is a speed threshold.
10. The computer readable medium of claim 9 , wherein the method further comprises:
after generating the one or more first commands, generating one or more second commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a second head motion threshold.
11. The computer readable medium of claim 8 , wherein the method further comprises:
after generating the one or more first commands, generating one or more third commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a third head motion threshold.
12. The computer readable medium of claim 8 , wherein the method further comprises:
after generating the one or more first commands, generating one or more fourth commands to move the OOI;
wherein the one or more fourth commands are generated in accordance to the motions of the user's head when the user's eye gaze moves by less than a first eye gaze motion threshold; and
wherein the one or more fourth commands are generated in accordance to the user's eye gaze when the user's eye gaze moves by greater than or equal to the first eye gaze motion threshold.
13. The computer readable medium of claim 12 , wherein the method further comprises:
stop generating the one or more fourth commands when the motions of the user's head fall under a fourth head motion threshold.
14. The computer readable medium of claim 8 , wherein the method further comprises:
after generating the one or more first commands, generating one or more fifth commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a fifth head motion threshold or until the user's eye gaze moves by greater than or equal to a second eye gaze motion threshold.
15. A system for a user to control an electronic device, the system comprising:
a processor configured to:
receive head signals indicative of motion of the user's head;
receive eye gaze signals indicative of the user's eye gaze;
when the motions of the user's head reach or cross a first head motion threshold to go outside a head motion tolerance zone, generate one or more first commands for the electronic device to move an object of interest (OOI) in accordance to the user's eye gaze; and
when the motions of the user's head stay within the head motion tolerance zone, not generate the one or more first commands to move the OOI in accordance to the user's eye gaze;
wherein the OOI is affected by the electronic device.
16. The system of claim 15 , wherein the head signals are indicative of speed of the user's head and wherein the first head motion threshold is a speed threshold.
17. The system of claim 16 , wherein after generation of the one or more first commands, the processor generates one or more second commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a second head motion threshold.
18. The system of claim 15 , wherein after generation of the one or more first commands, the processor generates one or more third commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a third head motion threshold.
19. The system of claim 15 , wherein after generation of the one or more first commands, the processor generates one or more fourth commands to move the OOI;
wherein the one or more fourth commands are generated in accordance to the motions of the user's head when the user's eye gaze moves by less than a first eye gaze motion threshold; and
wherein the one or more fourth commands are generated in accordance to the user's eye gaze when the user's eye gaze moves by greater than or equal to the first eye gaze motion threshold.
20. The system of claim 19 , wherein the processor stops generation of the one or more fourth commands when the motions of the user's head fall under a fourth head motion threshold.
21. The system of claim 15 , wherein after generation of the one or more first commands, the processor generates one or more fifth commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a fifth head motion threshold or until the user's eye gaze moves by greater than or equal to a second eye gaze motion threshold.
22. The system of claim 15 , wherein the system further comprises one or more first sensors that provide at least a portion of at least one of the head signals and the eye gaze signals.
23. The system of claim 22 , wherein at least one part of the system is worn on the user's body.
24. The system of claim 15 , wherein at least one part of the system is worn on the user's body.
25. An apparatus for a user to control an electronic device, the apparatus comprising:
a processor configured to:
receive head signals indicative of motion of the user's head;
receive eye gaze signals indicative of the user's eye gaze;
when the motions of the user's head reach or cross a first head motion threshold to go outside a head motion tolerance zone, generate a one or more first commands for the electronic device to move an object of interest (OOI) in accordance to the user's eye gaze;
when the motions of the user's head stay within the head motion tolerance zone, not generate the one or more first commands to move the OOI in accordance to the user's eye gaze; and
one or more first sensors configured to provide at least a portion of at least one of the head signals and the eye gaze signals;
wherein the OOI is affected by the electronic device.
26. The apparatus of claim 25 , wherein the head signals are indicative of speed of the user's head and wherein the first head motion threshold is a speed threshold.
27. The apparatus of claim 26 , wherein after generation of the one or more first commands, the processor generates one or more second commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a second head motion threshold.
28. The apparatus of claim 25 , wherein after generation of the one or more first commands, the processor generates one or more third commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a third head motion threshold.
29. The apparatus of claim 25 , wherein after generation of the one or more first commands, the processor generates one or more fourth commands to move the OOI;
wherein the one or more fourth commands are generated in accordance to the motions of the user's head when the user's eye gaze moves by less than a first eye gaze motion threshold; and
wherein the one or more fourth commands are generated in accordance to the user's eye gaze when the user's eye gaze moves by greater than or equal to the third eye gaze motion threshold.
30. The apparatus of claim 29 , wherein the processor stops generation of the one or more fourth commands when the motions of the user's head fall under a fourth head motion threshold.
31. The apparatus of claim 25 , wherein after generation of the one or more first commands, the processor generates one or more fifth commands to move the OOI in accordance to the motions of the user's head until the motions of the user's head fall under a fifth head motion threshold or until when the user's eye gaze moves by greater than or equal to a second eye gaze motion threshold.
32. The apparatus of claim 25 , wherein the apparatus is worn on the user's body.
33. The apparatus of claim 25 , wherein the apparatus is in the form of eyewear.
34. The apparatus of claim 25 , wherein the one or more first sensors comprise an image sensor.
35. The apparatus of claim 25 , where the one or more first sensors comprise a MEMS sensor.