Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions
User interaction concepts, principles and algorithms for gestures involving facial expressions, motion or orientation of body parts, eye gaze, tightening muscles, mental activity, and other user actions are disclosed. User interaction concepts, principles and algorithms for enabling hands-free and voice-free interaction with electronic devices are disclosed. Apparatuses, systems, computer implementable methods, and non-transient computer storage media storing instructions, implementing the disclosed concepts, principles and algorithms are disclosed. Gestures for systems using eye gaze and head tracking that can be used with augmented, mixed or virtual reality, mobile or desktop computing are disclosed. Use of periods of limited activity and consecutive user actions in orthogonal axes is disclosed. Generation of command signals based on start and end triggers is disclosed. Methods for coarse as well as fine modification of objects are disclosed.
1. A system for a user to control an electronic device, the system comprising:
at least one sensor configured to provide a portion of at least one of
i) head information (Head Info) indicative of at least one of motion and position of the user's head,
ii) eye gaze information (Eye Info) indicative of at least one of motion and position of the user's eye gaze, and
iii) facial expression (FE) information (FE Info) indicative of at least one of motion or position of one or more parts of the user's face; and
at least one processor configured to
i) start a first detection step, wherein the first detection step detects at least one of
a. the Head Info reaching or crossing a first head threshold, wherein the first head threshold is an amount of motion of the user's head or a position of the user's head,
b. the Eye Info reaching or crossing a first eye gaze threshold, wherein the first eye gaze threshold is an amount of motion of the user's eye gaze or a position of the user's eye gaze, and
c. a first FE becoming active for at least a first minimum FE detection time, wherein the first FE is said to be active when the FE Info reaches or crosses a first FE threshold and stays at or beyond the first FE threshold,
ii) after successful completion of the first detection step, start generation of command signals based on at least one of the Head Info and the Eye Info,
iii) after the start generation of command signals, start a second detection step, wherein the second detection step detects at least one of
a. the Head Info is within a second head threshold for at least a minimum head period of limited activity (POLA) time duration, wherein the second head threshold is less than the first head threshold and wherein the minimum head POLA time duration is greater than zero,
b. the Eye Info is within a second eye threshold for at least a minimum eye gaze POLA time duration, wherein the second eye gaze threshold is less than the first eye gaze threshold and wherein the minimum eye gaze POLA time duration is greater than zero, and
c. a second FE becoming active for at least a second minimum FE detection time, wherein the second FE is said to be active when the FE Info reaches or crosses a second FE threshold and stays at or beyond the second FE threshold, and wherein the second FE becomes active only after the first FE is no longer active, and
iv) after successful completion of the second detection step, stop generation of the command signals.
2. The system of claim 1 , wherein the command signals modify an object of interest (OOI).
3. The system of claim 2 , wherein the system further comprises a display screen, wherein the display screen displays the OOI.
4. The system of claim 3 , wherein the display screen is mounted on the user's head.
5. The system of claim 3 , wherein the OOI is a mouse pointer.
6. The system of claim 2 , wherein the processor is further configured to provide feedback to the user based on at least one of the first head threshold, the first eye gaze threshold, the first minimum FE detection time, the second head threshold, the second eye gaze threshold, the minimum head POLA time duration, the minimum eye POLA time duration, and the second minimum FE detection time.
7. The system of claim 2 , wherein at least one of the first head threshold, the first eye gaze threshold, the first minimum FE detection time, the second head threshold, the second eye gaze threshold, the minimum head POLA time duration, the minimum eye POLA time duration, and the second minimum FE detection time is settable by the user.
8. The system of claim 2 , wherein the at least one sensor is worn on or inside the user's body.
9. The system of claim 1 , wherein the at least one sensor comprises an image sensor.
10. The system of claim 1 , wherein the at least one sensor comprises a MEMS sensor.
11. The system of claim 1 , wherein the at least one sensor comprises an eye tracking sensor.
12. The system of claim 1 , wherein the at least one sensor is worn on or inside the user's body.
13. A computer implemented method of controlling an electronic device by a user, the computer implemented method comprising:
receiving at least one of
i) head information (Head Info) indicative of at least one of motion and position of the user's head,
ii) eye gaze information (Eye Info) indicative of at least one of motion and position of the user's eye gaze, and
iii) facial expression (FE) information (FE Info) indicative of at least one of motion or position of one or more parts of the user's face;
monitoring the received information for detection of a first event based on detecting at least one of
i) the Head Info reaching or crossing a first head threshold, wherein the first head threshold is an amount of motion of the user's head or a position of the user's head,
ii) the Eye Info reaching or crossing a first eye gaze threshold, wherein the first eye gaze threshold is an amount of motion of the user's eye gaze or a position of the user's eye gaze, and
iii) a first FE becoming active for at least a first minimum FE detection time, wherein the first FE is said to be active when the FE Info reaches or crosses a first FE threshold and stays at or beyond the first FE threshold;
after successful detection of the first event, start generating command signals based on at least one of the Head Info and the Eye Info;
after the start of generating command signals, start monitoring the received information for detection of second event wherein the second event is detected when at least one of
i) the Head Info is within a second head threshold for at least a minimum head period of limited activity (POLA) time duration, wherein the second head threshold is less than the first head threshold and wherein the minimum head POLA time duration is greater than zero,
ii) the Eye Info is within a second eye threshold for at least a minimum eye gaze POLA time duration, wherein the second eye gaze threshold is less than the first eye gaze threshold and wherein the minimum eye gaze POLA time duration is greater than zero, and
iii) a second FE becoming active for at least a second minimum FE detection time, wherein the second FE is said to be active when the FE info reaches or crosses a second FE threshold and stays at or beyond the second FE threshold, and wherein the second FE becomes active only after the first FE is no longer active; and
after detection of the end trigger, stop generating the command signals.
14. The computer-implemented method of claim 13 , wherein the command signals modify an object of interest (OOI) affected by the electronic device.
15. The computer-implemented method of claim 14 , wherein the OOI is a virtual object.
16. The computer-implemented method of claim 13 , wherein at least one part of the electronic device is worn on the user's body.
17. The computer-implemented method of claim 13 , wherein a MEMS sensor provides at least a portion of at least one of the Head Info, the Eye Info and the FE Info.
18. The computer-implemented method of claim 13 , wherein an image sensor provides at least a portion of at least one of the Head Info, the Eye Info and the FE Info.
19. A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:
receiving at least one of
i) head information (Head Info) indicative of at least one of motion and position of the user's head,
ii) eye gaze information (Eye Info) indicative of at least one of motion and position of the user's eye gaze, and
iii) facial expression (FE) information (FE Info) indicative of at least one of motion or position of one or more parts of the user's face;
detecting a first event based on at least one of
i) the Head Info reaching or crossing a first head threshold, wherein the first head threshold is an amount of motion of the user's head or a position of the user's head,
ii) the Eye Info reaching or crossing a first eye gaze threshold, wherein the first eye gaze threshold is an amount of motion of the user's eye gaze or a position of the user's eye gaze, and
iii) a first FE becoming active for at least a first minimum FE detection time, wherein the first FE is said to be active when the FE info reaches or crosses a first FE threshold and stays at or beyond the first FE threshold;
after successful detection of the first event, start generating command signals based on at least one of the Head Info and the Eye Info;
after the start generating of command signals, start detection of second event wherein the second event is detected when at least one of
i) the Head Info is within a second head threshold for at least a minimum head period of limited activity (POLA) time duration, wherein the second head threshold is less than the first head threshold and wherein the minimum head POLA time duration is greater than zero,
ii) the Eye Info is within a second eye threshold for at least a minimum eye gaze POLA time duration, wherein the second eye gaze threshold is less than the first eye gaze threshold and wherein the minimum eye gaze POLA time duration is greater than zero, and
iii) a second FE is becoming active for at least a second minimum FE detection time, wherein the second FE is said to be active when the FE info reaches or crosses a second FE threshold and stays at or beyond the second FE threshold, and wherein the second FE becomes active only after the first active FE is no longer active; and
after successful detection of the second event, stop the generating of the command signals.
20. The non-transitory computer readable medium of claim 19 , wherein the command signals modify an object of interest (OOI) affected by the electronic device.
21. The non-transitory computer readable medium of claim 20 , wherein the OOI is a virtual object.
22. The non-transitory computer readable medium of claim 19 , wherein at least one part of the electronic device is worn on the user's body.
23. The non-transitory computer readable medium of claim 19 , wherein a MEMS sensor provides at least a portion of at least one of the Head Info, the Eye Info and the FE Info.
24. The non-transitory computer readable medium of claim 19 , wherein an image sensor provides at least a portion of at least one of the Head Info, the Eye Info and the FE Info.