Augmented reality enabled command management
The exemplary embodiments disclose a method, a computer program product, and a computer system for managing user commands. The exemplary embodiments may include a user giving one or more commands to one or more devices, collecting data of the one or more commands, extracting one or more features from the collected data, and determining which one or more of the commands should be executed on which one or more of the devices based on the extracted one or more features and one or more models.
1. A computer-implemented method for managing user commands, the method comprising:
detecting a plurality of commands by way of a trigger;
collecting data of the commands given in succession by a user to interact with one or more devices of plural smart devices in response to detecting the plurality of commands;
extracting one or more features of the one or more devices of the plural smart devices from the collected data;
determining which of the commands should be executed on which one or more of the devices of the plural smart devices, based on the extracted one or more features, wherein the extracted one or more features are weighted to have a greater correlation with the determined one or more of the devices which correspond to the commands than a correlation between remaining features and the determined one or more of the devices which correspond to the commands;
and executing the commands on the one or more of the devices.
2. The method of claim 1 , further comprising:
notifying the user of which of the commands should be executed on which one or more of the devices.
3. The method of claim 1 , further comprising:
receiving feedback indicative of whether the determination of which commands should be executed on which one or more devices is appropriate; and
adjusting one or more machine learning models based on the received feedback.
4. The method of claim 3 , further comprising:
collecting training data from user interactions with respective ones of the plural smart devices;
extracting training features from the training data; and
training the one or more machine learning models based on the extracted training features.
5. The method of claim 1 , further comprising:
grouping two or more devices together, wherein the two or more grouped devices are user configured groups and the two or more grouped devices all execute any command that is directed towards any of the two or more devices.
6. The method of claim 1 , wherein the one or more features include trigger words, device names, command executability, pointing, waving, and eye direction.
7. A computer program product for managing user commands, the computer program product comprising:
one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of performing a method, the method comprising:
detecting a plurality of commands by way of a trigger;
collecting data of the commands given in succession by a user to interact with one or more devices of plural smart devices, wherein the collected data comprises audio data and video data, the video data being received from at least one sensor on an augmented reality (AR) device worn by the user in response to detecting the plurality of commands;
extracting one or more features of the one or more devices of the plural smart devices from the collected data;
determining which of the commands should be executed on which one or more of the devices based on the extracted one or more features, wherein the extracted one or more features are weighted to have a greater correlation with the determined one or more of the devices which correspond to the commands than a correlation between remaining features and the determined one or more of the devices which correspond to the commands;
and executing the commands on the one or more of the devices.
8. The computer program product of claim 7 , wherein the one or more features comprise a speech feature, a first gesture feature indicating a direction the user is pointing, and a second gesture feature indicating an eye direction of the user.
9. The computer program product of claim 7 , further comprising:
notifying the user of which of the commands should be executed on which one or more of the devices, wherein the notifying comprises displaying information on an overlay of the AR device utilized to affirm the information, the information indicating which of the commands should be executed on which one or more of the devices.
10. The computer program product of claim 7 , further comprising:
correlating, utilizing one or more models, the one or more features with the likelihood of determining which one or more of the devices the commands are directed towards.
11. The computer program product of claim 10 , further comprising:
receiving feedback indicative of whether the determination of which commands should be executed on which one or more devices is appropriate; and
adjusting the one or more models based on the received feedback.
12. The computer program product of claim 10 , further comprising:
collecting training data from user interactions with respective ones of the plural smart devices, wherein the training data is labelled training data that indicates the respective ones of the plural smart devices;
extracting training features from the training data; and
training the one or more models based on the extracted training features.
13. The computer program product of claim 7 , further comprising:
grouping two or more devices together, wherein the two or more grouped devices all execute any command that is directed towards any of the two or more devices.
14. A computer system for managing user commands, the computer system comprising:
one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more processors capable of performing a method, the method comprising:
detecting a plurality of commands by way of a trigger;
collecting data of the commands given in succession by a user to interact with one or more of plural smart devices in response to detecting the plurality of commands;
extracting features of one or more of the plural smart devices from the collected data, wherein the extracted features comprise a speech feature and a gesture feature;
determining, using the extracted features, which one or more of the plural smart devices each of the commands should be executed on, wherein the extracted features are weighted to have a greater correlation with the determined one or more of the plural smart devices which correspond to the commands than a correlation between remaining features and the determined one or more of the plural smart devices which correspond to the commands;
and executing the commands on the determined one of the plural smart devices.
15. The computer system of claim 14 , further comprising:
notifying the user, via an augmented reality interface, which one of the plural smart devices one or more of the commands should be executed on.
16. The computer system of claim 14 , further comprising:
correlating, utilizing one or more machine learning models, the extracted features with the likelihood of determining which one of the plural smart devices one or more of the commands is directed towards.
17. The computer system of claim 16 , further comprising:
receiving feedback indicative of whether the determination of which one of the plural smart devices is appropriate; and
adjusting the one or more machine learning models based on the received feedback.
18. The computer system of claim 16 , wherein the determining which one of the plural smart devices one or more of the commands should be executed on comprises applying the one or more machine learning models to the extracted features, comprising the speech feature and the gesture feature, to determine which one of the plural smart devices the user command is directed to.