IP Library Granted Patent US 11,803,247
Granted Patent B2
US 11,803,247 · App. 17/509,568 · Granted Oct 31, 2023

Gesture-based control of plural devices in an environment

Inventors: Mauro Marzorati (Lutz, FL); Todd Russell Whitman (Bethany, CT); Jeremy R. Fox (Georgetown, TX); Sarbajit K. Rakshit (Kolkata, IN)
Assignee: KYNDRYL, INC.
G06F3/017G06F3/016G06F9/451
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,803,247
App. No.
17/509,568
Granted
Oct 31, 2023
Kind
B2
Abstract

A computer-implemented method includes: predicting, by a computing device, devices for inclusion in an interface; generating, by the computing device, the interface including areas corresponding to the devices; detecting, by the computing device, a user selection of one of the areas of the interface; detecting, by the computing device, a hand gesture associated with the selected one of the areas; and transmitting, by the computing device, data defining the hand gesture to a respective one of the devices corresponding to the selected one of the areas, thereby causing the respective one of the devices to execute a command based on the hand gesture.

Claims (65)

1. A method, comprising:

predicting, by a hub computing device, devices for inclusion in an interface, the predicted devices being a subset of plural devices in a smart home environment, wherein each of the plural devices is configured to execute functions in response to receiving commands defined in terms of gestures;

generating, by the hub computing device, the interface including areas corresponding to the devices;

cause a smartwatch worn by a user to display the interface;

map the areas to respective physical spaces;

detecting, by the hub computing device, a hand gesture of the user in one of the physical spaces using data from a sensor in the smartwatch; and

transmitting, by the hub computing device, data defining the hand gesture to a respective one of the devices corresponding to the one of the physical spaces, thereby causing the respective one of the devices to execute a command based on the hand gesture, wherein the transmitting comprises transmitting the data from the hub computing device to the respective one of the devices.

2. The method of claim 1 , wherein the predicting is based on context.

3. The method of claim 1 , wherein the predicting is based on historical usage.

4. The method of claim 1 , wherein the predicting is based on a combination of context and historical usage.

5. The method of claim 1 , further comprising:

receiving user input to merge two of the areas of the interface into a merged area, the two of the areas comprising a first area associated with a first one of the devices and a second area associated with a second one of the devices;

re-generating the interface showing the merged area;

detecting a user selection of the merged area;

detecting another hand gesture associated with merged area; and

transmitting data defining the other hand gesture to the first device and the second device, thereby causing the first device and the second device to execute respective commands each based on the other hand gesture.

6. The method of claim 1 , further comprising:

receiving user input to split one of the areas of the interface into two sub-areas;

re-generating the interface showing the two sub-areas;

detecting a user selection of one of the two sub-areas;

detecting another hand gesture associated with the selected one of the two sub-areas;

transmitting data defining the other hand gesture to one of the devices corresponding to the selected one of the two sub-areas, thereby causing a sub-system of the one of the devices to execute a command based on the other hand gesture.

7. The method of claim 1 , further comprising:

receiving user input, from a first user device, defining customization of the interface;

storing data defining the customization of the interface;

receiving input from a second user device;

generating a customized interface using the data defining the customization of the interface; and

pushing the customized interface to the second user device.

8. The method of claim 1 , further comprising providing feedback to the user based on the detecting the hand gesture.

9. The method of claim 8 , wherein the feedback is haptic.

10. The method of claim 8 , wherein the feedback is visual in the interface.

11. A computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to cause a hub computing device to:

predict devices for inclusion in an interface, the predicted devices being a subset of plural devices in a smart home environment, wherein each of the plural smart devices is configured to execute functions in response to receiving commands defined in terms of gestures;

generate the interface including areas corresponding to the devices;

cause a smartwatch worn by a user to display the interface;

map the areas to respective physical spaces;

detect a gesture of a hand, of the user, in one of the physical spaces using data from a sensor in the smartwatch; and

transmit data defining the gesture to a respective one of the devices corresponding to the one of the physical spaces, thereby causing the respective one of the devices to execute a command based on the gesture, wherein the transmitting comprises transmitting the data from the hub computing device to the respective one of the devices.

12. The computer program product of claim 11 , wherein:

a total number of the plural smart devices exceeds a number of spaces in the interface; and

the predicting is based on:

context that is based on an on and off state of each respective one of the plural smart devices; and

historical usage that is based on how the user has interacted with one or more of the plural smart devices in the past.

13. The computer program product of claim 11 , wherein the program instructions are executable to provide feedback to the user based on detecting the hand in different ones of the respective physical spaces.

14. The computer program product of claim 13 , wherein the feedback is haptic feedback emitted by the smartwatch.

15. The computer program product of claim 13 , wherein the feedback is visual in the interface displayed on the smartwatch.

16. A system comprising:

a processor, a computer readable memory, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to cause a hub computing device to:

predict devices for inclusion in an interface, the predicting being based on a combination of context and historical usage of plural devices in a smart home environment, wherein the predicted devices comprise a subset of the plural devices in the smart home environment, and wherein each of the plural devices is configured to execute functions in response to receiving commands defined in terms of gestures;

generate the interface including areas corresponding to the predicted devices;

cause a smartwatch worn by a user to display the interface;

map the areas to respective physical spaces;

detect a gesture of a hand, of the user, in one of the physical spaces corresponding to one of the areas using data from a sensor in the smartwatch; and

transmit data defining the gesture to a respective one of the devices corresponding to the one of the physical spaces, thereby causing the respective one of the devices to execute a command based on the gesture, wherein the transmitting comprises transmitting the data from the hub computing device to the respective one of the devices.

17. The system of claim 16 , wherein the program instructions are executable to:

receive user input to merge two of the areas; and

re-generate the interface showing a merged area in place of the two of the areas.

18. The system of claim 16 , wherein the program instructions are executable to:

receive user input to split one of the areas; and

re-generate the interface showing two sub-areas in place of the one of the areas.

19. The system of claim 16 , wherein the program instructions are executable to:

receive user input defining a customization of the interface;

store data defining the customization;

generate a customized interface using the data defining the customization; and

automatically push the customized interface to a guest device.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 21, 2022
From: INTERNATIONAL BUSINESS MACHINES CORPORATION
To: KYNDRYL, INC.
Reel/Frame 061163/0097 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2021
From: MARZORATI, MAURO; WHITMAN, TODD RUSSELL; FOX, JEREMY R.; RAKSHIT, SARBAJIT K.
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 057901/0713 →
Continuity (1)
Related Publication 20230131489A1 · Apr 27, 2023