Systems and methods for alternative control of touch-based devices
A computer-implemented method, a system, and software includes providing output from a touch-based device to an external display; detecting gestures from a user located away from and not physically touching the touch-based device; and translating the detected gestures into appropriate commands for the touch-based device. The systems and methods provide alternative control of touch-based devices such as mobile devices. The systems and methods can include a mobile device coupled to an external display device and controlled via user gestures monitored by a collocated sensor. Accordingly, the systems and methods allow users to operate applications (“apps”) on the mobile device displayed on the external display device and controlled without touching the mobile device using gestures monitored by the collocated sensor. This enables the wide variety of rich apps to be operated in a new manner.
1. A computer-implemented method, comprising:
providing output from a touch-based device to an external display connected to the touch-based device;
detecting, via a sensor external from the touch-based device and directly facing a user and monitoring the user's body, gestures comprising in-air hand movements and in-air body movements from a user located away from and not physically touching the touch-based device; and
translating the detected gestures into appropriate touch commands based on the in-air hand movements and appropriate tilt commands based on the in-air body movements for the touch-based device to control an application on the touch-based device without physically interacting with the touch-based device, wherein the application is configured to be controlled by the detected gestures, and wherein the detected gestures are utilized in lieu of actual touch commands on the touch-based device and actual tilt commands through an accelerometer on the touch-based device without modifying the application, and wherein the translating comprises:
mapping the in-air hand movements to touch gestures meaningful to the touch-based device for the touch commands; and
mapping the in-air body movements to accelerometer readings meaningful to the touch-based device for the tilt commands.
2. The computer-implemented method of claim 1 , further comprising:
modeling user behavior to provide assistance in mapping the gestures to the appropriate commands for the touch-based device; and
constraining the gestures based on the modeling and settings.
3. The computer-implemented method of claim 1 , further comprising:
locating one or more body parts of the user; and
recognizing the gestures based on the location or locations of the one or more body parts.
4. The computer-implemented method of claim 1 , further comprising:
presenting an outline of the user or a cursor on the external display.
5. The computer-implemented method of claim 1 , wherein the touch-based device is input into a dock to perform the providing, the detecting, and the translating, wherein the dock is communicatively coupled to the external display and to a sensor for the detecting.
6. A system, comprising:
a network interface, a data store, and a processor, each communicatively coupled therebetween; and
memory storing instructions that, when executed, cause the processor to:
provide output from a touch-based device to an external display connected to the touch-based device;
detect, via a sensor external from the touch-based device and directly facing a user and monitoring the user's body, gestures comprising in-air hand movements and in-air body movements from a user located away from and not physically touching the touch-based device; and
translate the detected gestures into appropriate touch commands for the touch-based device based on the in-air hand movements and appropriate tilt commands based on the in-air body movements to control an application on the touch-based device without physically interacting with the touch-based device, wherein the application is configured to be controlled by the detected gestures, and wherein the detected gestures are utilized in lieu of actual touch commands on the touch-based device and actual tilt commands through an accelerometer on the touch-based device without modifying the application, wherein the detected gestures are translated by:
mapping the in-air hand movements to touch gestures meaningful to the touch-based device for the touch commands; and
mapping the in-air body movements to accelerometer readings meaningful to the touch-based device for the tilt commands.
7. The system of claim 6 , wherein the instructions, when executed, further cause the processor to:
modeling user behavior to provide assistance in mapping the gestures to the appropriate commands for the touch-based device; and
constraining the gestures based on the modeling and settings.
8. The system of claim 6 , wherein the instructions, when executed, further cause the processor to:
locating one or more body parts of the user; and
recognizing the gestures based on the location or locations of the one or more body parts.
9. The system of claim 6 , wherein the instructions, when executed, further cause the processor to:
present an outline of the user or a cursor on the external display.
10. A non-transitory computer readable medium with software stored thereon, the software comprising instructions executable by a system, and in response to such execution causes the system to perform operations comprising:
providing output from a touch-based device to an external display connected to the touch-based device;
detecting, via a sensor external from the touch-based device and directly facing a user and monitoring the user's body, gestures comprising in-air hand movements and in-air body movements from a user located away from and not physically touching the touch-based device; and
translating the detected gestures into appropriate touch commands based on the in-air hand movements and appropriate tilt commands based on the in-air body movements for the touch-based device to control an application on the touch-based device without physically interacting with the touch-based device, wherein the application is configured to be controlled by the detected gestures, and wherein the detected gestures are utilized in lieu of actual touch commands on the touch-based device and actual tilt commands through an accelerometer on the touch-based device without modifying the application, and wherein the translating comprises:
mapping the in-air hand movements to touch gestures meaningful to the touch-based device for the touch commands; and
mapping the in-air body movements to accelerometer readings meaningful to the touch-based device for the tilt commands.
11. The non-transitory computer readable medium of claim 10 , wherein the instructions executable by the system, and in response to such execution further causes the system to perform operations comprising:
modeling user behavior to provide assistance in mapping the gestures to the appropriate commands for the touch-based device; and
constraining the gestures based on the modeling and settings.
12. The non-transitory computer readable medium of claim 10 , wherein the instructions executable by the system, and in response to such execution further causes the system to perform operations comprising:
locating one or more body parts of the user; and
recognizing the gestures based on the location or locations of the one or more body parts.
13. The non-transitory computer readable medium of claim 10 , wherein the instructions executable by the system, and in response to such execution further causes the system to perform operations comprising:
presenting an outline of the user or a cursor on the external display.