System and method for incorporating gesture and voice recognition into a single system
A system and method for combining two separate types of human machine interfaces, e.g., a voice signal and a gesture signal, performing voice recognition to a voice signal and gesture recognition to the gesture signal. Based on a confidence determination using the voice recognition result and the gesture recognition result the system can, for example, immediately perform the command/request, request confirmation of the command/request or determine that the command/request was not identified.
1. A computer based method for executing a command in a device, comprising the steps of:
receiving a voice signal;
receiving a gesture signal;
performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;
performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;
determining a selected command, said selected command being one of the first command or the second command, based upon the first and second VR scores and said first and second GR scores; and
executing, in the device, said selected command.
2. The method of claim 1 , wherein said step of determining a selected command comprises:
generating a first combined recognition score for said first command based upon said first VR score and said first GR score;
generating a second combined recognition score for said second command based upon said second VR score and said second GR score; and
identifying said selected command based on said first and second combined recognition scores.
3. The method of claim 2 , wherein said step of identifying said selected command comprises
identifying said first command as said selected command if said first combined recognition score exceeds said second combined recognition score; and
identifying said second command as said selected command if said second combined recognition score exceeds said first combined recognition score.
4. A computer based method for executing a command in a device, comprising the steps of:
receiving a voice signal;
receiving a gesture signal;
performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;
performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;
determining a selected command, said selected command being one of the first command or the second command, based upon said first VR score and said second GR score; and
executing, in the device, said selected command.
5. The method of claim 4 , wherein said step of determining a selected command comprises:
generating a query for asking which of said first and second command to perform when said first VR score exceeds a first threshold and said second GR score exceeds a second threshold.
6. The method of claim 4 , wherein said query is at least one of an audible request and a visual query.
7. The method of claim 4 , wherein said step of determining a selected command comprises:
outputting to a user interface, a confirmation request that said first command should be executed, prior to executing said first command, when said first VR score exceeds a first threshold and said second GR score does not exceeds a second threshold.
8. The method of claim 7 , wherein said confirmation request is at least one of an audible request and a visual request.
9. The method of claim 4 , wherein said step of determining a selected command comprises:
outputting to a user interface, a confirmation request that said second command should be executed, prior to executing said second command, when said first VR score does not exceed a first threshold and said second GR score exceeds a second threshold.
10. The method of claim 9 , wherein said confirmation request is at least one of an audible request and a visual request.
11. A non-transitory computer readable medium comprising computer executable code which when executed by a computer performs the steps of:
receiving a voice signal;
receiving a gesture signal;
performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;
performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;
determining a selected command, said selected command being one of the first command or the second command, based upon the first and second VR scores and said first and second GR scores; and
executing, in a device, said selected command.
12. The non-transitory computer readable medium of claim 11 , wherein said step of determining a selected command comprises:
generating a first combined recognition score for said first command based upon said first VR score and said first GR score;
generating a second combined recognition score for said second command based upon said second VR score and said second GR score; and
identifying said selected command based on said first and second combined recognition scores.
13. The non-transitory computer readable medium of claim 12 , wherein said step of identifying said selected command comprises
identifying said first command as said selected command if said first combined recognition score exceeds said second combined recognition score; and
identifying said second command as said selected command if said second combined recognition score exceeds said first combined recognition score.
14. A non-transitory computer readable medium comprising computer executable code which when executed by a computer performs the steps of:
receiving a voice signal;
receiving a gesture signal;
performing a voice recognition procedure to generate a first voice recognition (VR) score representing a likelihood that the voice signal corresponds to a first command and to generate a second VR score representing a likelihood that the voice signal corresponds to a second command;
performing a gesture recognition procedure to generate a first gesture recognition (GR) score representing a likelihood that the gesture signal corresponds to said first command, and to generate a second GR score representing a likelihood that the gesture signal corresponds to said second command;
determining a selected command, said selected command being one of the first command or the second command, based upon said first VR score and said second GR score; and
executing, in a device, said selected command.
15. The non-transitory computer readable medium of claim 14 , wherein said step of determining a selected command comprises:
generating a query for asking which of said first and second command to perform when said first VR score exceeds a first threshold and said second GR score exceeds a second threshold.
16. The non-transitory computer readable medium of claim 14 , wherein said query is at least one of an audible request and a visual query.
17. The non-transitory computer readable medium of claim 14 , wherein said step of determining a selected command comprises:
outputting to a user interface, a confirmation request that said first command should be executed, prior to executing said first command, when said first VR score exceeds a first threshold and said second GR score does not exceeds a second threshold.
18. The non-transitory computer readable medium of claim 17 , wherein said confirmation request is at least one of an audible request and a visual request.
19. The non-transitory computer readable medium of claim 14 , wherein said step of determining a selected command comprises:
outputting to a user interface, a confirmation request that said second command should be executed, prior to executing said second command, when said first VR score does not exceed a first threshold and said second GR score exceeds a second threshold.
20. The non-transitory computer readable medium of claim 19 , wherein said confirmation request is at least one of an audible request and a visual request.