Systems, methods, apparatuses, computer readable medium for controlling electronic devices
View Patent ↗This application includes disclosure of methods, systems, apparatuses as well as principles/algorithms that can be implemented on computer readable medium, for defining user gestures, interpreting user actions, communicating and confirming user intent when communicating with electronic devices. A system for controlling an electronic device by a user is disclosed that includes a microprocessor and a communication link. The microprocessor runs control software for receiving a first signal indicative of a first action of the user, receives a second signal indicative of motion or position of a part of the user's body, and generates a command signal for the electronic device based on a user gesture performed by the user. The communication link communicates the command signal to the electronic device.
1. A system for a user to control an electronic device, the system comprising:
a microprocessor configured to:
receive head signals indicative of at least one of motion and position of the user's head;
receive eye gaze signals indicative of the user's eye gaze; and
generate control signals for the electronic device to move an object of interest (OOI) affected by the electronic device;
wherein the control signals are generated in accordance to only one of the head signals and the eye gaze signals depending on the eye gaze signals, such that the control signals to move the OOI are generated in accordance to the head signals and not the eye gaze signals when the user's eye gaze is detected and the eye gaze signals indicate a change in the user's eye gaze that is less than an eye gaze motion threshold, and the control signals to move the OOI are generated in accordance to the eye gaze signals and not the head signals when the user's eye gaze is detected and the eye gaze signals indicate a change in the user's eye gaze that is greater than or equal to the eye gaze motion threshold; wherein the eye gaze motion threshold is a specified amount of motion of the user's eye gaze and the eye gaze motion threshold is different than zero; and
wherein motion of the OOI resulting from a given amount of displacement performed by the user by using their head is less than motion of the OOI resulting from the given amount of displacement performed by the user by using their eye gaze.
2. The system of claim 1 , wherein the microprocessor is configured to receive third signals indicative of magnitude of at least one facial expression of the user, wherein the magnitude of the at least one facial expression of the user is determined based on motion or position of at least one facial muscle of the user.
3. The system of claim 1 , wherein the OOI comprises a graphical object affected by the electronic device.
4. The system of claim 1 , wherein the OOI comprises a mouse pointer.
5. The system of claim 1 , further comprising a first sensor, wherein the first sensor provides at least one of the head signals and the eye gaze signals.
6. The system of claim 5 , wherein the first sensor comprises an image sensor.
7. The system of claim 5 , wherein the first sensor comprises an inertial sensor.
8. The system of claim 5 , wherein the first sensor comprises an eye gaze tracking system.
9. The system of claim 5 , wherein the OOI comprises a graphical object affected by the electronic device.
10. The system of claim 5 , wherein the OOI comprises a mouse pointer.
11. The system of claim 2 , wherein the control signals are not generated when the third signals indicate the magnitude of the at least one facial expression is less than or equal to a facial expression threshold.
12. The system of claim 1 , wherein the change in the user's eye gaze is calculated independent of the location of the OOI.
13. The system of claim 1 , wherein a head gain curve relates the at least one of motion and position of the user's head to motion of the OOI resulting from the control signals generated in accordance to the head signals, an eye gaze gain curve relates motion of the user's eye gaze to motion of the OOI resulting from the control signals generated in accordance to the eye gaze signals, and the head gain curve is flatter than the eye gaze gain curve.
14. A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:
receiving head signals indicative of at least one of motion and position of the user's head;
receiving eye gaze signals indicative of the user's eye gaze; and
generating control signals for the electronic device to move an object of interest (OOI) affected by the electronic device,
wherein the control signals are generated in accordance to only one of the head signals and the eye gaze signals depending on the eye gaze signals, such that the control signals to move the OOI are generated in accordance to the head signals and not the eye gaze signals when the user's eye gaze is detected and the eye gaze signals indicate a change in the user's eye gaze that is less than an eye gaze motion threshold, and the control signals to move the OOI are generated in accordance to the eye gaze signals and not the head signals when the user's eye gaze is detected and the eye gaze signals indicate a change in the user's eye gaze that is greater than or equal to the eye gaze motion threshold; wherein the eye gaze motion threshold is a specified amount of motion of the user's eye gaze and the eye gaze motion threshold is different than zero; and
wherein motion of the OOI resulting from a given amount of displacement performed by the user by using their head is less than or equal to motion of the OOI resulting from the given amount of displacement performed by the user by using their eye gaze.
15. The non-transitory computer readable medium of claim 14 , wherein the method further comprises receiving third signals indicative of magnitude of at least one facial expression of the user, wherein the magnitude of the at least one facial expression of the user is determined based on motion or position of at least one facial muscle of the user.
16. The non-transitory computer readable medium of claim 15 , wherein the control signals are not generated when the third signals indicate the magnitude of the at least one facial expression is less than or equal to a facial expression threshold.
17. The non-transitory computer readable medium of claim 14 , wherein the change in the user's eye gaze is calculated independent of the location of the OOI.
18. The non-transitory computer readable medium of claim 14 , wherein a head gain curve relates the at least one of motion and position of the user's head to motion of the OOI resulting from the control signals generated in accordance to the head signals, an eye gaze gain curve relates motion of the user's eye gaze to motion of the OOI resulting from the control signals generated in accordance to the eye gaze signals, and the head gain curve is flatter than the eye gaze gain curve.
19. An apparatus for a user to control an electronic device, the apparatus comprising:
memory;
a microprocessor configured to:
receive head signals indicative of at least one of motion and position of the user's head;
receive eye gaze signals indicative of the user's eye gaze; and
generate control signals to move an object of interest (OOI) affected by the electronic device; and
a first sensor configured to provide at least one of the head signals and the eye gaze signals;
wherein the control signals are generated in accordance to only one of the head signals and the eye gaze signals depending on the eye gaze signals, such that the control signals to move the OOI are generated in accordance to the head signals and not the eye gaze signals when the user's eye gaze is detected and the eye gaze signals indicate a change in the user's eye gaze that is less than an eye gaze motion threshold, and the control signals to move the OOI are generated in accordance to the eye gaze signals and not the head signals when the user's eye gaze is detected and the eye gaze signals indicate a change in the user's eye gaze that is greater than or equal to the eye gaze motion threshold; wherein the eye gaze motion threshold is a specified amount of motion of the user's eye gaze and the eye gaze motion threshold is different than zero; and
wherein motion of the OOI resulting from a given amount of displacement performed by the user by using their head is less than or equal to motion of the OOI resulting from the given amount of displacement performed by the user by using their eye gaze.
20. The apparatus of claim 19 , wherein the OOI comprises a graphical object affected by the electronic device.
21. The apparatus of claim 19 , wherein the OOI comprises a mouse pointer.
22. The apparatus of claim 19 , wherein the apparatus is part of the electronic device.
23. The apparatus of claim 19 , wherein the microprocessor is further configured to receive third signals indicative of magnitude of at least one facial expression of the user, wherein the magnitude of the at least one facial expression of the user is determined based on motion or position of at least one facial muscle of the user.
24. The apparatus of claim 19 , wherein the first sensor comprises an image sensor.
25. The apparatus of claim 19 , wherein the first sensor comprises an inertial sensor.
26. The apparatus of claim 19 , wherein the first sensor comprises an eye gaze tracking system.
27. The apparatus of claim 22 , wherein the OOI comprises a graphical object affected by the electronic device.
28. The apparatus of claim 22 , wherein the OOI comprises a mouse pointer.
29. The apparatus of claim 19 , wherein said apparatus is worn on the user's body.
30. The apparatus of claim 23 , wherein the control signals to move the OOI are not generated when the third signals indicate the magnitude of the at least one facial expression is less than or equal to a facial expression threshold.
31. The apparatus of claim 19 , wherein the change in the user's eye gaze is calculated independent of the location of the OOI.
32. The apparatus of claim 19 , wherein a head gain curve relates the at least one of motion and position of the user's head to motion of the OOI resulting from the control signals generated in accordance to the head signals, an eye gaze gain curve relates motion of the user's eye gaze to motion of the OOI resulting from the control signals generated in accordance to the eye gaze signals, and the head gain curve is flatter than the eye gaze gain curve.
33. A computer implemented method for a user to communicate with an electronic device, said computer implemented method comprising:
receiving head signals indicative of at least one of motion and position of the user's head, wherein the head signals are provided by a first sensor;
receiving eye gaze signals indicative of the user's eye gaze, wherein the eye gaze signals are provided by a second sensor; and
generating control signals to move an object of interest OOI affected by the electronic device,
wherein the control signals are generated in accordance to only one of the head signals and the eye gaze signals depending on the eye gaze signals, such that the control signals to move the OOI are generated in accordance to the head signals and not the eye gaze signals when the user's eye gaze is detected by the second sensor and the eye gaze signals indicate a change in the user's eye gaze that is less than a first eye gaze motion threshold, and the control signals to move the OOI are generated in accordance to the eye gaze signals and not the head signals when the user's eye gaze is detected by the second sensor and the eye gaze signals indicate a change in the user's eye gaze that is greater than or equal to a second eye gaze motion threshold; wherein the first and second eye gaze motion thresholds are each a specified amount of motion of the user's eye gaze and the first and second eye gaze motion thresholds are different than zero; and
wherein motion of the OOI resulting from a given amount of displacement performed by the user by using their head is less than motion of the OOI resulting from the given amount of displacement performed by the user by using their eye gaze.
34. The computer implemented method of claim 33 , further comprising:
receiving third signals indicative of magnitude of at least one facial expression of the user, wherein the magnitude of the at least one facial expression of the user is determined based on motion or position of at least one facial muscle of the user.
35. The computer-implemented method of claim 33 , wherein the first eye gaze motion threshold is equal to the second eye gaze motion threshold.
36. The computer implemented method of claim 34 , further comprising:
not generating the control signals when the third signals indicate the magnitude of the at least one facial expression of the user is less than or equal to a facial expression threshold.
37. The computer implemented method of claim 33 , wherein the change in the user's eye gaze is calculated independent of the location of the OOI.
38. The computer implemented method of claim 33 , wherein a head gain curve relates the at least one of motion and position of the user's head to motion of the OOI resulting from the control signals generated in accordance to the head signals, an eye gaze gain curve relates motion of the user's eye gaze to motion of the OOI resulting from the control signals generated in accordance to the eye gaze signals, and the head gain curve is flatter than the eye gaze gain curve.