Computer interaction method, device, and program product
Embodiments of the present disclosure provide a computer interaction method, device, and program product. The method includes: acquiring, in response to triggering of an input to an electronic device, multiple images that present a given part of a user; determining a corresponding character sequence based on respective gestures of the given part in the multiple images, corresponding characters in the character sequence being selected from a predefined character set in which multiple characters respectively correspond to different gestures of the given part; and determining, based on the character sequence, a computer instruction to be input to the electronic device. With this solution, the user can conveniently and flexibly execute the input to the electronic device through a gesture of the given part (e.g., a hand).
1. A computer interaction method, including:
acquiring, in response to triggering of an input to an electronic device, multiple images that present a given part of a user, the multiple images comprising a first image and a second image, the first image being associated with a first character based on a first gesture of the given part in the multiple images, and the second image being associated with a second character based on a second gesture of the given part in the multiple images;
determining a corresponding character sequence based on at least the first character and the second character associated with the first and second images in the multiple images, the first and second characters in the character sequence being selected from a predefined character set in which multiple characters respectively correspond to different gestures of the given part; and
determining, based on the character sequence of at least the first and second characters, a computer instruction to be input to the electronic device;
wherein determining the computer instruction includes:
correcting the character sequence of at least the first and second characters using a trained language model; and
determining a computer instruction corresponding to the corrected character sequences;
wherein the predefined character set includes:
multiple natural language characters respectively corresponding to multiple gestures of the given part; and
at least one special symbol; and
wherein at least one gesture corresponding to the at least one special symbol is different from any of the multiple gestures respectively corresponding to the multiple natural language characters.
2. The method according to claim 1 , wherein determining the character sequence includes:
recognizing the respective gestures of the given part in the multiple images using a trained multi-modal recognition model; and
determining the character sequence corresponding to the recognized respective gestures based on a correspondence between corresponding characters in the predefined character set and the respective gestures of the given part.
3. The method according to claim 2 , wherein the multi-modal recognition model is compressed in a training process by means of model pruning or parameter quantization.
4. The method according to claim 1 , wherein the predefined character set includes at least a sign language character set which includes at least a portion of the multiple natural language characters respectively corresponding to multiple gestures of a hand.
5. The method according to claim 1 , wherein the gestures corresponding to the at least one special symbol being are different from any of the multiple gestures respectively corresponding to the multiple natural language characters.
6. The method according to claim 5 , wherein the at least one special symbol includes at least one of a space symbol, an asterisk, and a slash symbol.
7. The method according to claim 4 , wherein the sign language character set includes multiple English letters in American Sign Language (ASL) or multiple phonetic characters in Chinese Sign Language.
8. The method according to claim 1 , wherein the computer instruction includes at least one of an input related to user verification and an input of a computer-executable command.
9. An electronic device, including:
at least one processor; and
at least one memory storing computer-executable instructions, the at least one memory and the computer-executable instructions being configured to cause, together with the at least one processor, the electronic device to perform actions including:
acquiring, in response to triggering of an input to the electronic device, multiple images that present a given part of a user, the multiple images comprising a first image and a second image, the first image being associated with a first character based on a first gesture of the given part in the multiple images, and the second image being associated with a second character based on a second gesture of the given part in the multiple images;
determining a corresponding character sequence based on at least the first character and the second character associated with the first and second images in the multiple images, the first and second characters in the character sequence being selected from a predefined character set in which multiple characters respectively correspond to different gestures of the given part; and
determining, based on the character sequence of at least the first and second characters, a computer instruction to be input to the electronic device;
wherein determining the computer instruction includes:
correcting the character sequence of at least the first and second characters using a trained language model; and
determining a computer instruction corresponding to the corrected character sequences;
wherein the predefined character set includes:
multiple natural language characters respectively corresponding to multiple gestures of the given part; and
at least one special symbol; and
wherein at least one gesture corresponding to the at least one special symbol is different from any of the multiple gestures respectively corresponding to the multiple natural language characters.
10. The device according to claim 9 , wherein determining the character sequence includes:
recognizing the respective gestures of the given part in the multiple images using a trained multi-modal recognition model; and
determining the character sequence corresponding to the recognized respective gestures based on a correspondence between corresponding characters in the predefined character set and the respective gestures of the given part.
11. The device according to claim 10 , wherein the multi-modal recognition model is compressed in a training process by means of model pruning or parameter quantization.
12. The device according to claim 9 , wherein the predefined character set includes at least a sign language character set which includes at least a portion of the multiple natural language characters respectively corresponding to multiple gestures of a hand.
13. The device according to claim 9 , wherein the gestures corresponding to the at least one special symbol are different from any of the multiple gestures respectively corresponding to the multiple natural language characters.
14. The device according to claim 13 , wherein the at least one special symbol includes at least one of a space symbol, an asterisk, and a slash symbol.
15. The device according to claim 12 , wherein the sign language character set includes multiple English letters in American Sign Language (ASL) or multiple phonetic characters in Chinese Sign Language.
16. The device according to claim 9 , wherein the computer instruction includes at least one of an input related to user verification and an input of a computer-executable command.
17. A computer program product that is tangibly stored on a non-volatile computer-readable medium and includes computer-executable instructions, wherein the computer-executable instructions, when executed, cause a device to:
acquire, in response to triggering of an input to an electronic device, multiple images that present a given part of a user, the multiple images comprising a first image and a second image, the first image being associated with a first character based on a first gesture of the given part in the multiple images, and the second image being associated with a second character based on a second gesture of the given part in the multiple images;
determine a corresponding character sequence based on at least the first character and the second character associated with the first and second images in the multiple images, the first and second characters in the character sequence being selected from a predefined character set in which multiple characters respectively correspond to different gestures of the given part; and
determine, based on the character sequence of at least the first and second characters, a computer instruction to be input to the electronic device;
wherein determining the computer instruction includes:
correcting the character sequence of at least the first and second characters using a trained language model; and
determining a computer instruction corresponding to the corrected character sequences;
wherein the predefined character set includes:
multiple natural language characters respectively corresponding to multiple gestures of the given part; and
at least one special symbol; and
wherein at least one gesture corresponding to the at least one special symbol is different from any of the multiple gestures respectively corresponding to the multiple natural language characters.
18. The computer program product according to claim 17 , wherein determining the character sequence includes:
recognizing the respective gestures of the given part in the multiple images using a trained multi-modal recognition model; and
determining the character sequence corresponding to the recognized respective gestures based on a correspondence between corresponding characters in the predefined character set and the respective gestures of the given part.
19. The computer program product according to claim 17 , wherein the predefined character set includes at least a sign language character set which includes at least a portion of the multiple natural language characters respectively corresponding to multiple gestures of a hand.
20. The computer program product according to claim 17 , wherein the gestures corresponding to the at least one special symbol are different from any of the multiple gestures respectively corresponding to the multiple natural language characters.