Systems and methods for extensions to alternative control of touch-based devices
Systems and methods configured to facilitate multi-modal user inputs in lieu of physical input for a processing device configured to execute an application include obtaining non-physical input for processing device and the application, wherein the physical input comprises one or more of touch-based input and tilt input; processing the non-physical input to convert into appropriate physical input commands for the application; and providing the physical input commands to the processing device.
1. A method configured to facilitate multi-modal user inputs in lieu of physical input for a processing device configured to execute an application, the method comprising:
obtaining non-physical input for the processing device and the application, wherein the physical input comprises one or more of touch-based input and tilt input;
processing the non-physical input to convert into appropriate physical input commands for the application;
providing the physical input commands to the processing device; and
enhancing menu screens in the application on the processing device to assist the user from getting to gameplay, wherein the enhancing comprises any of implementing a macro and modeling the menu screens on a graph to find a shortest path to the gameplay.
2. The method of claim 1 , wherein the non-physical input comprises voice commands used to initiate multiple actions in the application, wherein the voice commands are converted into the appropriate physical input commands.
3. The method of claim 1 , wherein the non-physical input comprises input from a game controller which is converted into the appropriate physical input commands.
4. The method of claim 1 , wherein the non-physical input comprises input from a handheld remote which is converted into the appropriate physical input commands.
5. The method of claim 1 , wherein the application is controlled by physical touch commands on a touch screen and accelerometer readings associated with moving the touch-based device.
6. The method of claim 1 , wherein the non-physical input comprises gestures detected by a depth sensor which is converted into the appropriate physical input commands.
7. The method of claim 6 , wherein video and/or image data used to detect the gestures is from an imaging device associated with the processing device.
8. The method of claim 6 , wherein video and/or image data used to detect the gestures is from a sensor communicatively coupled to the processing device.
9. The method of claim 1 , wherein the processing comprises:
tracking hand position of the user in video and/or image data;
detecting a pose of the hand position through the video and/or image data by assigning a pose to the hand for at least one of the images from a set of potential poses, the set of potential poses include a first active pose and a null pose;
in response to detecting the first active pose, generating a control input command for the application; and
in response to the null pose, refraining from generation of any control input command for the application.
10. The method of claim 9 , wherein the null pose is based on proximity of the hand position to at least one of one or more other body parts of the subject and one or more foreign objects.
11. The method of claim 1 , further comprising:
utilizing a “snap to” technique on menu screens for the application on the processing device where a cursor enters a region of interest and is snapped to an anchor which translates large gesture movements into bounded, fine grained cursor movements.
12. A system configured to facilitate gesture-based user inputs in lieu of physical input for a processing device configured to execute an application, the system comprising:
an interface communicatively coupled to the processing device; and
a processor communicatively coupled to the processing device and configured to
obtain non-physical input for processing device and the application, wherein the physical input comprises one or more of touch-based input and tilt input;
process the non-physical input to convert into appropriate physical input commands for the application;
provide the physical input commands to the processing device; and
enhance menu screens in the application on the processing device to assist the user from getting to gameplay, wherein the enhance comprises any of implementing a macro and modeling the menu screens on a graph to find a shortest path to the gameplay.
13. The system of claim 12 , wherein the non-physical input comprises voice commands used to initiate multiple actions in the application, wherein the voice commands are converted into the appropriate physical input commands.
14. The system of claim 12 , wherein the non-physical input comprises input from a game controller which is converted into the appropriate physical input commands.
15. The system of claim 12 , wherein the non-physical input comprises input from a handheld remote which is converted into the appropriate physical input commands.
16. The system of claim 12 , wherein the application is controlled by physical touch commands on a touch screen and accelerometer readings associated with moving the touch-based device.
17. The system of claim 12 , wherein the non-physical input comprises gestures detected by a depth sensor which is converted into the appropriate physical input commands.
18. A method configured to facilitate multi-modal user inputs in lieu of physical input for a processing device configured to execute an application, the method comprising:
obtaining non-physical input for the processing device and the application, wherein the physical input comprises one or more of touch-based input and tilt input;
processing the non-physical input to convert into appropriate physical input commands for the application, wherein the processing comprises:
tracking hand position of the user in video and/or image data;
detecting a pose of the hand position through the video and/or image data by assigning a pose to the hand for at least one of the images from a set of potential poses, the set of potential poses include a first active pose and a null pose;
in response to detecting the first active pose, generating a control input command for the application; and
in response to the null pose, refraining from generation of any control input command for the application; and
providing the physical input commands to the processing device.
19. The method of claim 18 , wherein the non-physical input comprises voice commands used to initiate multiple actions in the application, wherein the voice commands are converted into the appropriate physical input commands.
20. The method of claim 18 , wherein the non-physical input comprises input from a game controller or a handheld remote which is converted into the appropriate physical input commands.