Enhanced virtual touchpad
A method, including receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface and segmenting the physical surface into one or more physical regions. A functionality is assigned to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, and a sequence of three-dimensional (3D) maps is received, the sequence of 3D maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions. The 3D maps are analyzed to detect a gesture performed by the user, and based on the gesture, an input is simulated for the tactile input device corresponding to the one of the physical regions.
1. A method, comprising:
presenting, by a computer, multiple interactive items on a display coupled to the computer;
projecting a light toward a scene that includes a user of the computer;
capturing and processing the projected light returned from the scene so as to reconstruct an initial three-dimensional (3D) map;
capturing and processing a two-dimensional (2D) image containing reflections from an eye of the user;
obtaining 3D coordinates of the head of the user based on the initial 3D map;
identifying, based on the 3D coordinates of the head and the reflections from the eye, a direction of a gaze of the user;
detecting, in response to the gaze direction, that the user is gazing toward an area of the display; and
in a gesture-based interaction step, subsequent to detecting that the user is gazing toward the display:
receiving a sequence of three-dimensional (3D) maps containing at least a hand of the user;
analyzing the 3D maps to detect an operation performed by the user, in which the user performs a first finger gesture, then moves the finger side-to-side in a requested scroll direction, and then performs a second finger gesture;
in response to the operation performed by the user, scrolling the interactive items on the display in the requested scroll direction by an amount that depends on a length of side-to-side movement of the finger and a distance of the user from the multiple interactive items on the display; and
controlling a speed of the scrolling in response to the direction of the gaze of the user.
2. The method according to claim 1 , and comprising analyzing the 3D maps to detect a Press and Hold gesture and presenting, on the display, context information for the selected interactive item in response to the Press and Hold gesture.
3. The method according to claim 1 , and comprising analyzing the 3D maps to detect a Tap gesture.
4. The method according to claim 1 , and comprising analyzing the 3D maps to detect a Swipe gesture or a Select gesture and task switching to an application associated with a selected interactive item in response to the detected gesture being selected from a list consisting of the Swipe gesture and the Select gesture.
5. The method according to claim 1 , wherein the detected operation comprises a Pinch gesture, in which two or more fingers of the hand move toward each other.
6. The method according to claim 1 , and comprising analyzing the 3D maps to detect a Pinch gesture or a Grab gesture and changing, on the display, a size of a selected interactive item in response to the detected gesture being selected from a list consisting of the Pinch gesture and the Grab gesture.
7. The method according to claim 1 , and further comprising switching between executing applications in response to detecting a Swipe From Edge gesture.
8. The method according to claim 1 , and further comprising presenting, on the display, a hidden menu in response to detecting a Swipe From Edge gesture.
9. The method according to claim 1 , and comprising presenting, on the display, a rotation of the selected interactive item in response to detecting a Rotate gesture.
10. The method according to claim 1 , and further comprising identifying a color of an object held by the hand of the user, and using the color for presenting content on the display.
11. An apparatus, comprising:
a display, which is configured to present multiple interactive items;
a sensing device, which is configured:
to project a light toward a scene that includes a user of the apparatus,
to capture and process the projected light returned from the scene so as to reconstruct an initial three-dimensional (3D) map,
to capture and process a two-dimensional (2D) image containing reflections from an eye of the user,
to obtain 3D coordinates of the head of the user based on the initial 3D map, and
to identify, based on the 3D coordinates of the head and the reflections from the eye, a direction of a gaze of the user; and
a computer, which is coupled to the sensing device and the display, and is configured:
to detect, in response to the gaze direction, that the user is gazing toward an area of the display, and
in a gesture-based interaction step, subsequent to detecting that the user is gazing toward the display, to receive a sequence of three-dimensional (3D) maps containing at least a hand of the user,
to analyze the 3D maps to detect an operation performed by the user, in which the user performs a first finger gesture, then moves the finger side-to-side in a requested scroll direction, and then performs a second finger gesture,
in response to the operation performed by the user, scrolling the interactive items on the display in the requested scroll direction by an amount that depends on a length of side-to-side movement of the finger and a distance of the user from the multiple interactive items on the display, and
to control a speed of the scrolling in response to the direction of the gaze of the user.
12. The apparatus according to claim 11 , wherein the computer is configured to analyze the 3D maps to detect a Press and Hold gesture and to present, on the display, context information for the selected interactive item in response to the Press and Hold gesture.
13. The apparatus according to claim 11 , wherein the computer is configured to analyze the 3D maps to detect a Swipe gesture or a Select gesture and to task switch to an application associated with a selected interactive item in response to the detected gesture being selected from a list consisting of the Swipe gesture and the Select gesture.
14. The apparatus according to claim 11 , wherein the detected operation comprises a Pinch gesture, in which two or more fingers of the hand move toward each other.
15. The apparatus according to claim 11 , wherein the computer is configured to analyze the 3D maps to detect a Pinch gesture or a Grab gesture and to change, on the display, a size of the selected interactive item in response to the detected gesture being selected from a list consisting of the Pinch gesture and the Grab gesture.
16. The method according to claim 1 , wherein scrolling the interactive items comprises measuring the length of the side-to-side movement in the 3D maps and setting a number of pixels to scroll in proportion to the measured length.
17. The method according to claim 1 , wherein the amount to scroll is adjusted by a factor that compensates for distance of the user from the interactive items on the display.
18. The method according to claim 1 , wherein controlling the speed of the scrolling comprises stopping the scrolling.
19. The method according to claim 18 , wherein stopping the scrolling comprises stopping the scrolling in response to detecting that the user is gazing in a first direction and moving the finger in a second direction.
20. The method according to claim 1 , wherein controlling the speed of the scrolling comprises interpolating between respective speeds of eye movement and of finger movement.