Multi-device gesture control
Aspects of the subject technology relate to gesture-control inputs to an electronic device for controlling one or more other devices. The electronic device can efficiently provide gesture control for multiple other devices by mapping a finite set of user gestures to a specific set of gesture-control elements for each of the multiple other devices. In this way a single gesture can be detected for potentially controlling various different functions of various different devices. Prior to gesture control, the electronic device may receive a selection of a particular one of the multiple other devices for control, and obtain the specific set of gesture-control elements for gesture control of that selected device.
1. A method, comprising:
determining, by an electronic device storing one or more gesture-control elements for each of a plurality of devices, that the electronic device is within a predetermined range of the plurality of devices;
receiving a selection, based on data from at least a first sensor of the electronic device, of one of the plurality of devices for gesture control;
obtaining, based on the selection, the one or more gesture-control elements for the one of the plurality of devices;
providing, for display, one or more gesture-control icons, respectively, for the one or more gesture-control elements for the one of the plurality of devices;
receiving, by the electronic device, a selection of one of the one or more gesture-control icons corresponding to one of the one or more gesture-control elements;
receiving sensor data from at least a second sensor of the electronic device;
recognizing one of a predetermined plurality of gestures by providing the received sensor data to a machine learning model trained to recognize each of the predetermined plurality of gestures based on input sensor data from at least the second sensor; and
controlling the one of the plurality of devices by applying the recognized one of the predetermined plurality of gestures to the one of the one or more gesture-control elements corresponding to the selected one of the one or more gesture-control icons.
2. The method of claim 1 , wherein the controlling comprises:
generating, by the electronic device, a control signal corresponding to the one of the one or more gesture-control elements; and
transmitting the control signal to the one of the plurality of devices.
3. The method of claim 1 , wherein receiving the sensor data from at least the second sensor of the electronic device comprises receiving the sensor data from at least the second sensor of the electronic device while the one of the plurality of devices is selected for control and while the one of the one or more gesture-control icons is selected.
4. The method of claim 1 , wherein receiving a selection of the one of the plurality of devices for gesture control comprises determining, using the data from the first sensor, that the electronic device is pointed at the one of the plurality of devices.
5. The method of claim 4 , further comprising, prior to determining that the electronic device is pointed at the one of the plurality of devices, and responsive to determining, by the electronic device, that the electronic device is within the predetermined range of the plurality of devices, that a user of the electronic device has engaged in a pointing gesture, and that the electronic device is pointed in the direction of the plurality of devices:
providing, for display, a plurality of device icons arranged for display according to a plurality of corresponding locations of the plurality of devices.
6. The method of claim 5 , further comprising highlighting the device icon for the one of the plurality of devices when the data from the first sensor indicates that the electronic device is pointed at the one of the plurality of devices.
7. The method of claim 5 , further comprising:
following the controlling, receiving a subsequent selection, based on additional data from at least the first sensor of the electronic device, of a different one of the plurality of devices for gesture control;
obtaining, based on the selection, the one or more gesture-control elements for the different one of the plurality of devices, wherein the one or more gesture-control elements for the one of the plurality of devices is different from one or more gesture-control elements for the different one of the plurality of devices; and
providing, for display in place of the one or more gesture-control icons for the one or more gesture-control elements for the one of the plurality of devices, one or more different gesture control icons corresponding to the one or more gesture-control elements for the different one of the plurality of devices.
8. The method of claim 4 , wherein the at least the second sensor comprises an electromyography sensor.
9. The method of claim 8 , wherein the at least the second sensor further comprises an inertial measurement unit.
10. The method of claim 9 , wherein the one or more gesture-control elements for the one of the plurality of devices comprise a plurality of gesture-control elements for the one of the plurality of devices.
11. The method of claim 9 , further comprising receiving, by the electronic device using the inertial measurement unit, a selection of one of the one or more gesture-control elements for the one of the plurality of devices.
12. The method of claim 11 , wherein recognizing the one of the predetermined plurality of gestures by providing the received sensor data to the machine learning model comprises providing electromyography data from the electromyography sensor to the machine learning model while the one of the one or more gesture-control elements for the one of the plurality of devices is selected.
13. The method of claim 8 , wherein the one of the one or more gesture-control elements corresponding to the selected one of the one or more gesture-control icons comprises a continuous-control element, and wherein recognizing the one of the predetermined plurality of gestures by providing the received sensor data to the machine learning model comprises generating a continuous control output from the machine learning model by providing a stream of electromyography data from the electromyography sensor to the machine learning model.
14. An electronic device, comprising:
a first sensor;
a second sensor;
memory storing:
one or more gesture-control elements for each of a plurality of devices; and
a machine learning model trained to recognize each of a predetermined plurality of gestures based on input sensor data from at least the second sensor; and
one or more processors configured to:
determine that the electronic device is within a predetermined range of the plurality of devices;
receive a selection, based on data from at least the first sensor of the electronic device, of one of the plurality of devices for gesture control;
obtain, based on the selection, the one or more gesture-control elements for the one of the plurality of devices;
provide, for display, one or more gesture-control icons, respectively, for the one or more gesture-control elements for the one of the plurality of devices;
receive a selection of one of the one or more gesture-control icons corresponding to one of the one or more gesture-control elements;
receive sensor data from at least the second sensor;
recognize one of the predetermined plurality of gestures by providing the received sensor data to the machine learning model; and
control the one of the plurality of devices by applying the recognized one of the predetermined plurality of gestures to the one of the one or more gesture-control elements corresponding to the selected one of the one or more gesture-control icons.
15. The electronic device of claim 14 , wherein the one or more processors are configured to receive a selection of the one of the plurality of devices for gesture control by determining, using the data from the first sensor, that the electronic device is pointed at the one of the plurality of devices.
16. The electronic device of claim 15 , wherein the one or more processors are further configured to, prior to determining that the electronic device is pointed at the one of the plurality of devices, and responsive to determining that the electronic device is within the predetermined range of the plurality of devices, that a user of the electronic device has engaged in a pointing gesture, and that the electronic device is pointed in the direction of the plurality of devices, provide, for display, a plurality of device icons arranged for display according to a plurality of corresponding locations of the plurality of devices.
17. The electronic device of claim 16 , wherein the one or more processors are further configured to highlight the device icon for the one of the plurality of devices when the data from the first sensor indicates that the electronic device is pointed at the one of the plurality of devices.
18. The electronic device of claim 15 , wherein the second sensor comprises at least one of an electromyography sensor, an accelerometer, a gyroscope, or a magnetometer.
19. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
determining, by an electronic device storing one or more gesture-control elements for each of a plurality of devices, that the electronic device is within a predetermined range of the plurality of devices;
receiving a selection, based on data from at least a first sensor of the electronic device, of one of the plurality of devices for gesture control;
obtaining, based on the selection, the one or more gesture-control elements for the one of the plurality of devices;
providing, for display, one or more gesture-control icons, respectively, for the one or more gesture-control elements for the one of the plurality of devices;
receiving, by the electronic device, a selection of one of the one or more gesture-control icons corresponding to one of the one or more gesture-control elements;
receiving sensor data from at least a second sensor of the electronic device;
recognizing one of a predetermined plurality of gestures by providing the received sensor data to a machine learning model trained to recognize each of the predetermined plurality of gestures based on input sensor data from at least the second sensor; and
controlling the one of the plurality of devices by applying the recognized one of the predetermined plurality of gestures to the one of the one or more gesture-control elements corresponding to the selected one of the one or more gesture-control icons.
20. The non-transitory machine-readable medium of claim 19 , wherein the one or more gesture-control elements for the one of the plurality of devices comprise a plurality of gesture-control elements for the one of the plurality of devices.
21. The non-transitory machine-readable medium of claim 20 , the operations further comprising receiving, by the electronic device using an inertial measurement unit of the electronic device, a selection of one of the plurality of gesture-control elements for the one of the plurality of devices.
22. The non-transitory machine-readable medium of claim 21 , wherein recognizing the one of the predetermined plurality of gestures by providing the received sensor data to the machine learning model comprises providing electromyography data from an electromyography sensor to the machine learning model while the one of the plurality of gesture-control elements for the one of the plurality of devices is selected.
23. A method, comprising:
determining, by an electronic device storing one or more gesture-control elements for each of a plurality of devices, that the electronic device is within a predetermined range of the plurality of devices;
obtaining, based on a selection of one of the plurality of devices using the electronic device, the one or more gesture-control elements for the one of the plurality of devices;
providing, for display, one or more gesture-control icons;
receiving, by the electronic device, a selection of one of the one or more gesture-control icons corresponding to one of the one or more gesture-control elements;
receiving sensor data from a sensor of the electronic device;
providing the sensor data to a machine learning model trained to recognize each of a predetermined plurality of gestures based on input sensor data from the sensor; and
controlling the one of the plurality of devices based on an output of the machine learning model.