Multipurpose controllers and methods
View Patent ↗Method and apparatus is disclosed for a user to communicate with an electronic device. A processor receives user intention actions comprising facial expression (FE) information indicative of facial expressions and body information indicative of motion or position of one or more body parts of the user. When the FE or body information crosses a first level, the processor starts generating first signals based on the FE or body information to communicate with the electronic device. When the FE or body information crosses a second level, the processor can end generation of the first signals or modify the first signals. An image processing or eye gaze tracking system can provide some FE information or body information. The signals can modify attributes of an object of interest. Use of thresholds that are independent of sensor position or orientation with respect to the user's body are also disclosed.
1. A system for a user to communicate with an electronic device, the system comprising:
one or more processors configured to receive facial expression information (FE Info) indicative of one or more facial actions of the user; and
one or more first sensors configured to provide at least a portion of said FE Info based on said one or more facial actions performed by the user;
wherein said one or more processors are configured to:
i) analyze FE Info to detect one or more first facial actions in progress;
ii) when said one or more first facial actions are detected to be in progress for a first minimum time duration, start repeated generation of one or more first command signals; and
iii) stop generation of said one or more first command signals when said one or more first facial actions are no longer detected to be in progress;
wherein said first minimum time duration is greater than zero; and
wherein said one or more facial actions of the user comprise use of one or more of a facial muscle, jaw, tongue, mouth, cheek, nose, brow, eyebrow, ear, and eyelid of the user.
2. The system of claim 1 , wherein at least one of said one or more first sensors is included in eye wear configured to be worn by the user.
3. The system of claim 1 , wherein at least one of said one or more first sensors is included in head wear configured to be worn by the user.
4. The system of claim 1 , wherein said one or more processors are further configured to change time interval between repetitions of generated said one or more first command signals based on a magnitude of said one or more first facial actions.
5. The system of claim 4 , further comprising:
one or more body sensors configured to provide body information (Body Info) indicative of at least one of motion and position of the user's one or more body parts;
wherein said one or more processors are further configured to receive said Body Info; and
wherein said one or more processors are further configured to generate one or more second signals based on said Body Info.
6. The system of claim 5 , wherein said one or more body sensors are configured to be placed at least partially inside a body cavity of the user.
7. The system of claim 4 , wherein said one or more first command signals comprise at least one of one or more zoom commands and one or more scroll commands.
8. The system of claim 1 , wherein said one or more first command signals comprise at least one of one or more zoom commands and one or more scroll commands.
9. The system of claim 1 , wherein said one or more first command signals modify an object of interest affected by the electronic device.
10. The system of claim 9 , wherein said object of interest is a virtual object on a display screen.
11. The system of claim 1 , wherein said one or more processors are further configured to:
(a) indicate said one or more first facial actions to be in progress when said FE Info crosses a first threshold;
(b) after step (a), continue to indicate said one or more first facial actions remain in progress until said FE Info crosses a second threshold, at which point indicate said one or more first facial actions to be no longer in progress; and
wherein said first threshold is different than said second threshold.
12. The system of claim 11 , wherein said second threshold is less than said first threshold.
13. The system of claim 11 , wherein said second threshold is greater than said first threshold.
14. The system of claim 1 , further comprising:
one or more body sensors configured to provide body information (Body Info) indicative of at least one of motion and position of the user's one or more body parts;
wherein said one or more processors are further configured to receive said Body Info; and
wherein said one or more processors are further configured to generate one or more second signals based on said Body Info.
15. A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to allow a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:
i) receiving facial expression information (FE Info) indicative of one or more facial actions performed by the user;
ii) analyzing said FE Info to detect one or more first facial actions in progress;
iii) when said one or more first facial actions are detected to be in progress for a first minimum time duration, start repeatedly generating one or more first command signals; and
iv) stop generating said one or more first command signals when said one or more first facial actions are no longer detected to be in progress;
wherein said first minimum time duration is greater than zero;
wherein said one or more facial actions of the user comprise use of one or more of a facial muscle, jaw, tongue, mouth, cheek, nose, brow, eyebrow, ear, and eyelid of the user; and
wherein at least a portion of said FE Info is provided by one or more first sensors.
16. The non-transitory computer readable medium of claim 15 , wherein said one or more body sensors are configured to be at least partially placed inside a body cavity of the user.
17. The non-transitory computer readable medium of claim 15 , wherein at least one of said one or more first sensors is included in eye wear configured to be worn by the user.
18. The non-transitory computer readable medium of claim 15 , wherein at least one of said one or more first sensors is included in head wear configured to be worn by the user.
19. The non-transitory computer readable medium of claim 15 , wherein said one or more first sensors comprise one or more image sensors configured to monitor part of the user's skin, and one or more radiation sources configured to illuminate said part of the user's skin.
20. The non-transitory computer readable medium of claim 15 , wherein said one or more processors are further configured to change time interval between repetitions of said repeatedly generated one or more first command signals based on a magnitude of said one or more first facial actions.
21. The non-transitory computer readable medium of claim 20 , wherein said one or more first command signals comprise at least one of one or more zoom commands and one or more scroll commands.
22. The non-transitory computer readable medium of claim 15 , wherein said one or more first command signals comprise at least one of one or more zoom commands and one or more scroll commands.
23. The non-transitory computer readable medium of claim 15 , wherein said one or more first command signals modify an object of interest affected by the electronic device.
24. The non-transitory computer readable medium of claim 23 , wherein said object of interest is a virtual object on a display screen.
25. The non-transitory computer readable medium of claim 15 , wherein said method further comprises:
(a) indicating said one or more first facial actions to be in progress when said FE Info crosses a first threshold; and
(b) after step (a), continue indicating said one or more first facial actions remain in progress until said FE Info crosses a second threshold, at which point indicating said one or more first facial actions to be no longer be in progress;
wherein said first threshold is different than said second threshold.
26. The non-transitory computer readable medium of claim 25 , wherein said second threshold is less than said first threshold.
27. The non-transitory computer readable medium of claim 25 , wherein said second threshold is greater than said first threshold.
28. The non-transitory computer readable medium of claim 15 , wherein said method further comprises:
receiving body information (Body Info) indicative of at least one of motion and position of the user's one or more body parts; and
generating one or more second signals based on said Body Info;
wherein one or more body sensors provide at least a portion of said Body Info.
29. An apparatus for a user to communicate with an electronic device, the apparatus comprising:
one or more processors configured to receive:
i) facial expression information (FE Info) indicative of one or more facial actions of the user, and
ii) body information (Body Info) indicative of at least one of motion and position of the user's one or more body parts;
one or more facial action sensors configured to provide at least a portion of said FE Info based on said one or more facial actions of the user;
one or more body sensors configured to provide at least a portion of said Body Info based on said at least one of motion and position of the user's one or more body parts;
wherein said one or more processors are configured to:
a) analyze FE Info to detect one or more first facial actions in progress;
b) when said one or more first facial actions are detected to be in progress for a first minimum time duration, start repeated generation of one or more first command signals; and
c) stop generation of said one or more first command signals when said one or more first facial actions are no longer detected to be in progress;
wherein said first minimum time duration is greater than zero;
wherein said one or more facial actions of the user comprise use of one or more of a facial muscle, jaw, tongue, mouth, cheek, nose, brow, eyebrow, ear, and eyelid of the user; and
wherein said one or more processors are further configured to generate one or more second signals based on said Body Info.
30. The apparatus of claim 29 , wherein said one or more processors are further configured to change time interval between repetitions of generated said one or more first command signals based on a magnitude of said one or more first facial actions.
31. The apparatus of claim 29 , wherein at least one of said one or more first command signals and said one or more second signals modify an object of interest affected by the electronic device.
32. The apparatus of claim 29 , wherein the apparatus is configured to be worn by the user.