Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions
User interaction concepts, principles and algorithms for gestures involving facial expressions, motion or orientation of body parts, eye gaze, tightening muscles, mental activity, and other user actions are disclosed. User interaction concepts, principles and algorithms for enabling hands-free and voice-free interaction with electronic devices are disclosed. Apparatuses, systems, computer implementable methods, and non-transient computer storage media storing instructions, implementing the disclosed concepts, principles and algorithms are disclosed. Gestures for systems using eye gaze and head tracking that can be used with augmented, mixed or virtual reality, mobile or desktop computing are disclosed. Use of periods of limited activity and consecutive user actions in orthogonal axes is disclosed. Generation of command signals based on start and end triggers is disclosed. Methods for coarse as well as fine modification of objects are disclosed.
1 . A system for a user to control an electronic device, the system comprising:
one or more first sensors configured to provide eye gaze information (Eye Info) indicative of the user's eye gaze;
one or more second sensors configured to provide facial action information (FA Info) indicative of one or more facial actions of the user;
one or more third sensors configured to provide head information (Head Info) indicative of motion or position of the user's head; and
one or more processors configured to:
i) receive said Eye Info, said FA Info and said Head Info;
ii) detect, based on said FA Info, a first facial action being performed by the user;
iii) upon detection of said first facial action, generate one or more first command signals to modify an object of interest (OOI) based on said Eye Info; and
iv) after generating said one or more first command signals, generate one or more second command signals to further modify said OOI based on said Head Info, said Eye Info, or both said Head Info and said Eye Info;
wherein said one or more second command signals are generated based on said Eye Info when a magnitude of change in eye gaze derived from said Eye Info is greater than or equal to an eye gaze change threshold; and
wherein said one or more second command signals are generated based on said Head Info when said magnitude of change in eye gaze derived from said Eye Info is less than said eye gaze change threshold.
2 . The system of claim 1 , wherein said one or more third sensors are configured to be worn on the user's head.
3 . The system of claim 1 , wherein said first facial action comprises an action involving one or more muscles of an eyelid, an eyebrow, or both.
4 . The system of claim 1 , wherein said first facial action comprises an action involving a nose, a mouth, a jaw, a lip, a cheek, or any combination thereof, of the user.
5 . The system of claim 1 , wherein said one or more first command signals are generated based on Eye Info received before detection of said first facial action.
6 . The system of claim 1 , wherein said one or more second sensors are configured to be worn on the user's head.
7 . The system of claim 1 , wherein said one or more processors are further configured to stop generation of said one or more second command signals when a head motion or position derived from said Head Info remains within a specified tolerance zone for at least a specified minimum time duration.
8 . The system of claim 1 , wherein said one or more processors are further configured to:
i. detect, based on said FA Info, a second facial action being performed by the user; and
ii. upon detection of said second facial action, stop generation of said one or more second command signals.
9 . The system of claim 1 ,
wherein said one or more processors are further configured to stop generation of said one or more second command signals when an eye gaze derived from said Eye Info remains within a specified tolerance zone for at least a specified minimum time duration.
10 . The system of claim 2 , wherein said one or more second sensors are configured to be worn on the user's head.
11 . The system of claim 1 , the system further comprising a display mechanism, wherein said display mechanism displays said OOI.
12 . The system of claim 11 , wherein said display mechanism is configured to be worn on the user's head.
13 . The system of claim 1 , wherein said first command signals modify at least one of location or appearance of said OOI.
14 . The system of claim 1 , wherein said one or more second sensors comprise an emitter configured to emit electromagnetic radiation towards the facial skin of the user.
15 . The system of claim 1 , wherein generating said one or more second command signals comprises applying a blending function to determine an amount of influence of said Eye Info and an amount of influence of said Head Info on the modification of said OOI.
16 . The system of claim 1 , wherein said one or more first sensors and said one or more second sensors comprise at least one common sensor configured to provide both said Eye Info and said FA Info.
17 . A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:
receiving eye gaze information (Eye Info) indicative of the user's eye gaze, head information (Head Info) indicative of motion or position of the user's head and facial action information (FA Info) indicative of one or more facial actions of the user;
detecting, based on said FA Info, a first facial action being performed by the user;
upon detection of said first facial action, generating one or more first command signals to modify an object of interest (OOI) based on said Eye Info; and
after generating said one or more first command signals, generating one or more second command signals to further modify said OOI based on said Head Info, said Eye Info, or both said Head Info and said Eye Info;
wherein said one or more second command signals are generated based on said Eye Info when a magnitude of change in eye gaze derived from said Eye Info is greater than or equal to an eye gaze change threshold; and
wherein said one or more second command signals are generated based on said Head Info when said magnitude of change in eye gaze derived from said Eye Info is less than said eye gaze change threshold.
18 . The non-transitory computer readable medium of claim 17 , wherein said Head Info is provided by one or more third sensors configured to be worn on the user's head.
19 . The non-transitory computer readable medium of claim 17 , wherein said first facial action comprises an action involving one or more muscles of an eyelid, an eyebrow, or both, of the user.
20 . The non-transitory computer readable medium of claim 17 , wherein said first facial action comprises an action involving a nose, a mouth, a jaw, a lip, a cheek, or any combination thereof, of the user.
21 . The non-transitory computer readable medium of claim 17 , wherein said one or more first command signals are generated based on Eye Info received before detection of said first facial action.
22 . The non-transitory computer readable medium of claim 17 , wherein said FA Info is provided by one or more second sensors configured to be worn on the user's head.
23 . The non-transitory computer readable medium of claim 17 , wherein said method further comprises:
stopping generation of said one or more second command signals when a head motion or position derived from said Head Info remains within a specified tolerance zone for at least a specified minimum time duration.
24 . The non-transitory computer readable medium of claim 17 , wherein said method further comprises: first command signals are based on Eye Info received at least one of:
i. detecting, based on said FA Info, a second facial action being performed by the user; and
ii. upon detection of said second facial action, stopping generation of said one or more second command signals.
25 . The non-transitory computer readable medium of claim 17 , wherein said method further comprises:
stopping generation of said one or more second command signals when an eye gaze derived from said Eye Info remains within a specified tolerance zone for at least a specified minimum time duration.
26 . The non-transitory computer readable medium of claim 17 , wherein
said Eye Info is provided by one or more first sensors configured to be worn on the user's head.
27 . The non-transitory computer readable medium of claim 17 , wherein generating said one or more second command signals comprises applying a blending function to determine an amount of influence of said Eye Info and an amount of influence of said Head Info on the modification of said OOI.
28 . A head-worn apparatus for a user, the apparatus comprising:
one or more first sensors configured to provide eye gaze information (Eye Info) indicative of the user's eye gaze;
one or more second sensors configured to provide facial action information (FA Info) indicative of one or more facial actions of the user;
one or more third sensors configured to provide head information (Head Info) indicative of motion or position of the user's head;
one or more display mechanisms; and
one or more processors configured to:
i) receive said Eye Info, said FA Info and said Head Info;
ii) detect, based on said FA Info, a first facial action being performed by the user;
iii) upon detection of said first facial action, generate one or more first command signals to modify an object of interest (OOI) based on said Eye Info, wherein said OOI is displayed using said one or more display mechanisms; and
iv) after generating said one or more first command signals, generate one or more second command signals to further modify said OOI based on said Head Info, said Eye Info, or both said Head Info and said Eye Info;
wherein said one or more second command signals are generated based on said Eye Info when a magnitude of change in eye gaze derived from said Eye Info is greater than or equal to an eye gaze change threshold; and
wherein said one or more second command signals are generated based on said Head Info when said magnitude of change in eye gaze derived from said Eye Info is less than said eye gaze change threshold.
29 . The head-worn apparatus of claim 28 , wherein said one or more processors are further configured to stop generation of said one or more second command signals when a head motion or position derived from said Head Info remains within a specified tolerance zone for at least a specified minimum time duration.
30 . The head-worn apparatus of claim 28 , wherein said one or more processors are further configured to:
i. detect, based on said FA Info, a second facial action being performed by the user; and
ii. upon detection of said second facial action, stop generation of said one or more second command signals.
31 . The head-worn apparatus of claim 28 ,
wherein said one or more processors are further configured to stop generation of said one or more second command signals when an eye gaze derived from said Eye Info remains within a specified tolerance zone for at least a specified minimum time duration.
32 . The head-worn apparatus of claim 28 , wherein said one or more display mechanisms comprise a display screen or a retinal projector.
33 . The head-worn apparatus of claim 28 , wherein said first facial action comprises blinking one or more eyes of the user.
34 . The head-worn apparatus of claim 28 , wherein said first facial action comprises an action involving one or more muscles of an eyelid, an eyebrow, or both, of the user.
35 . The head-worn apparatus of claim 28 , wherein said first facial action comprises an action involving a nose, a mouth, a jaw, a lip, a cheek, or any combination thereof, of the user.
36 . The head-worn apparatus of claim 28 , wherein generating said one or more second command signals comprises applying a blending function to determine an amount of influence of said Eye Info and an amount of influence of said Head Info on the modification of said OOI.
37 . The head-worn apparatus of claim 28 , wherein said one or more first sensors and said one or more second sensors comprise at least one common sensor configured to provide both said Eye Info and said FA Info.