Enhanced accessibility in mixed reality experience for collaboration tools
Described herein is are systems and methods for interpreting gesture(s) and/or sign(s) using a machine-learned model. Information regarding gesture(s) and/or sign(s) is received from a first user. The information can be received via a mixed reality device of the first user and/or a second user. Probabilities that the gesture(s) or sign(s) have particular meanings are calculated using a machine-trained model. The gesture(s) and/or sign(s) are interpreted in accordance with the calculated probabilities. Information regarding the interpreted gesture(s) and/or sign(s) are provided (e.g., displayed as visual text and/or an audible output) to the second user.
1. A system comprising:
a processor; and
a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:
receive information regarding a sequence of gestures or signs from a first user, the information received via a mixed reality device, the sequence including a first gesture or sign and a second gesture or sign, wherein the first gesture or sign occurs in the sequence before the second gesture or sign;
calculate probabilities that the second gesture or sign has particular meanings using a machine-trained model, the probabilities being based at least on the first gesture or sign that occurs in the sequence before the second gesture or sign;
interpret the second gesture or the sign in accordance with the calculated probabilities; and
provide information regarding the interpreted second gesture or sign to a second user.
2. The system of claim 1 , wherein the information regarding the interpreted second gesture or sign is provided as displayed text to the second user.
3. The system of claim 1 , wherein the information regarding the interpreted second gesture or sign is provided audibly to the second user.
4. The system of claim 1 , wherein the memory has further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:
translate the interpreted second gesture or sign to a target written language; and
provide information regarding the translated, interpreted second gesture or sign to the second user.
5. The system of claim 1 , wherein the machine-trained model comprises a classifier trained to recognize at least one of a plurality of gestures or signs of a particular sign language.
6. The system of claim 1 , wherein the machine-trained model is trained using a cluster algorithm in an unsupervised environment.
7. The system of claim 1 , wherein the memory has further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:
calculate the probabilities that the second gesture or sign has the particular meanings based at least on a third gesture or sign that occurs in the sequence after the second gesture or sign.
8. The system of claim 1 , wherein the memory has further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:
buffer the sequence of gestures or signs;
using the machine-trained model, calculate respective probabilities for each gesture or sign in the sequence;
interpret each gesture or sign in the sequence based at least on the respective probabilities; and
provide respective information regarding each interpreted gesture or sign in the sequence.
9. The system of claim 1 , further comprising the mixed reality device.
10. The system of claim 1 , wherein the machine-trained model is trained to recognize gestures or signs of a particular user for a particular sign language.
11. The system of claim 1 , wherein the memory has further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:
provide information regarding the interpreted second gesture or sign to the first user;
receive feedback from the first user regarding the interpreted second gesture or sign; and
update the machine-trained model in accordance with the feedback received from the first user.
12. The system of claim 1 , wherein the first gesture or sign and the second gesture or sign are from American Sign Language.
13. A method comprising:
receiving information regarding a sequence of gestures or signs from a first user, the information received via a mixed reality device, the sequence including a first gesture or sign and a second gesture or sign, wherein the first gesture or sign occurs in the sequence before the second gesture or sign;
based at least on the first gesture or sign, calculating probabilities that the second gesture or sign has particular meanings using a machine-trained model;
interpreting the second gesture or the sign in accordance with the calculated probabilities; and
providing information regarding the interpreted second gesture or sign to a second user.
14. The method of claim 13 , wherein the information regarding the interpreted second gesture or sign is provided audibly to the second user.
15. The method of claim 13 , further comprising:
translating the interpreted second gesture or sign to a target written language; and
providing information regarding the translated, interpreted second gesture or sign to the second user.
16. The method of claim 13 , wherein the machine-trained model is trained using a cluster algorithm in an unsupervised environment.
17. The method of claim 13 , wherein the information regarding the interpreted second gesture or sign is provided as displayed text to the second user.
18. The method of claim 13 , wherein the machine-trained model is trained to recognize gestures or signs of a particular user for a particular sign language.
19. A computer storage media storing computer-readable instructions that, when executed, cause a computing device to:
receive information regarding a particular gesture or sign from a first user, the information received via a mixed reality device, wherein at least one other gesture or sign is received sequentially with the particular gesture or sign;
calculate probabilities that the particular gesture or sign has particular meanings using a machine-trained model based at least on the at least one other gesture or sign received sequentially with the particular gesture or sign;
interpret the particular gesture or sign in accordance with the calculated probabilities; and
output information regarding the interpreted particular gesture or sign.
20. The computer storage media of claim 19 , storing further computer-readable instructions that, when executed, cause a computing device to:
translate the interpreted particular gesture or the sign to a target written language; and
provide information regarding the translated, interpreted particular gesture or sign to a second user.