IP Library Granted Patent US 9,971,490
Granted Patent B2
US 9,971,490 · App. 14/597,138 · Granted May 15, 2018

Device control

Inventors: Yinpeng Chen (Sammamish, WA); Zicheng Liu (Bellevue, WA); Zhengyou Zhang (Bellevue, WA)
Assignee: Microsoft Technology Licensing, LLC
G06F3/04847G06F3/017G06F3/0484G06F3/0488G06F3/04842G06F3/04883G06F21/32H04N9/04H04N13/0271G06F2203/04101
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,971,490
App. No.
14/597,138
Granted
May 15, 2018
Kind
B2
Abstract

The description relates to interactions with a display device. In one example, the interactions can include detecting a user proximate to a display and detecting a non-touch control gesture performed by the user proximate to the display. The example can also include presenting a graphical user interface (GUI) on the display that includes options associated with the control gesture. The example can also include receiving user input selecting one of the options and receiving additional user input from the user to interact with the GUI via the selected one of the options.

Claims (62)

1. A computing device comprising:

at least one processor; and

one or more computer-readable storage media having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

sense a user proximate to a display device;

identify the user utilizing one or more human recognition algorithms, at least one of the human recognition algorithms relating to sensed skeletal information;

locate the left hand and the right hand of the user from the sensed skeletal information;

detect a non-touch gesture performed by the user with either the left hand or the right hand and identify whether the user performed the non-touch gesture with the left hand or the right hand based at least upon the sensed skeletal information;

alter a graphical user interface on the display device based at least upon an identity of the user, the non-touch gesture, and whether the non-touch gesture was performed by the left hand or the right hand, the graphical user interface being altered by:

presenting a particular menu on the graphical user interface when the non-touch gesture is performed with the left hand; and

presenting content other than the particular menu when the non-touch gesture is performed with the right hand;

receive touch input from the user relative to the altered graphical user interface and determine whether the touch input was performed with the left hand or the right hand; and

further alter the graphical user interface based at least upon the received touch input and whether the received touch input was determined to have been performed by the left hand or the right hand.

2. The computing device of claim 1 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to:

in an instance when the non-touch gesture is performed with the left hand, further alter the graphical user interface by:

selecting a menu item from the particular menu based at least on the touch input; and

altering the graphical user interface based at least on the selected menu item.

3. A system, comprising:

a display device;

a first set of sensors configured to sense an area in front of the display device;

a second set of sensors configured to detect user input proximate to the display device; and

a processor configured to:

analyze first data from the first set of sensors to identify a user and non-touch gestures performed by the user;

when the first data indicates that the user performed an individual non-touch gesture, control a graphical user interface presented on the display device by:

presenting a particular menu on the display device when the individual non-touch gesture is performed with the left hand; and

presenting content other than the particular menu when the individual non-touch gesture is performed with the right hand;

when the particular menu is displayed, analyze second data from the second set of sensors to identify a touch gesture performed by the user with respect to the particular menu; and

further refine the control of the graphical user interface based at least on the second data by selecting a menu item from the particular menu and modifying the graphical user interface based at least on the selected menu item.

4. The system of claim 3 , wherein the second set of sensors are touch sensors integrated in the display device.

5. The system of claim 3 , wherein the first set of sensors are physically mounted on the display device.

6. The system of claim 3 , wherein the processor is further configured to:

receive user inputs selecting specific menu items to include on the particular menu; and

when the user is identified by analyzing the first data, include the specific menu items on the particular menu.

7. The system of claim 3 , manifest as a touch sensitive tablet type device or notebook computer type device where the second set of sensors is integrated with the display device as touch sensors.

8. The system of claim 3 , manifest as a large display device configured to be mounted on a wall or other vertical surface and wherein the first and second sets of sensors are integral to the large display device.

9. The system of claim 3 , wherein the first set of sensors is manifest as a set of red green blue+depth (RGBD) sensors and the second set of sensors is manifest as a set of touch sensors.

10. The system of claim 9 , wherein the touch sensors are configured to sense pressure or capacitance.

11. The system of claim 3 , wherein the processor is configured to calibrate the display device, the first set of sensors, and the second set of sensors by causing a number of objects to be presented in the graphical user interface and instructing the user to touch the objects while being sensed by the first set of sensors and the second set of sensors.

12. A computer-implemented method, comprising:

detecting a user proximate to a display;

detecting a non-touch gesture performed by the user;

determining whether the non-touch gesture is performed by the left hand or the right hand of the user;

presenting a particular menu on the display when the non-touch gesture is performed with the left hand;

presenting content other than the particular menu on the display when the non-touch gesture is performed with the right hand;

when the particular menu is displayed, receiving a touch input identifying a selected menu option from the particular menu;

receiving a further touch input from the user; and

controlling the display based at least upon the selected menu option and the further touch input.

13. The method of claim 12 , wherein the detecting the user proximate the display comprises identifying the user.

14. The method of claim 13 , wherein the identifying the user comprises detecting parameters.

15. The method of claim 14 , wherein the detected parameters include skeletal parameters, facial recognition parameters, and handedness parameters.

16. The method of claim 14 , wherein the identifying the user comprises comparing the detected parameters to stored parameters.

17. The method of claim 12 , further comprising:

receiving user input identifying specific menu items to include in the particular menu;

determining an identity of the user; and

presenting the specific menu items on the particular menu based at least on the identity of the user.

18. The method of claim 17 , further comprising:

capturing a depth image of the user; and

determining the identity of the user from the depth image.

19. The method of claim 12 , wherein the particular menu is a color palette, the selected menu option identifies a selected color, the further touch input comprises writing input, and the controlling comprises showing the writing input on the display in the selected color.

20. The method of claim 18 ,

further comprising:

detecting skeletal data of the user from the depth image; and

using the skeletal data to determine whether the user performed the non-touch gesture with the right hand or the left hand.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 14, 2015
From: CHEN, YINPENG; LIU, ZICHENG; ZHANG, ZHENGYOU
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 034716/0362 →
Continuity (2)
Provisional Application 61944917 · Feb 26, 2014
Related Publication 20150242107A1 · Aug 27, 2015