Computer vision based sign language interpreter
A system and method for translating sign language utterances into a target language, including: receiving motion capture data; producing phonemes/sign fragments from the received motion capture data; producing a plurality of sign sequences from the phonemes/sign fragments; parsing these sign sequences to produce grammatically parsed sign utterances; translating the grammatically parsed sign utterances into grammatical representations in the target language; and generating output utterances in the target language based upon the grammatical representations.
1. A method of translating sign language utterances into a target language, comprising:
receiving motion capture data of a signing user making signs in the sign language;
determining one or more manual sign features in parallel with one or more eye markers based on an independent temporal segmentation of the motion capture data for the manual sign features and the one or more eye markers;
producing phonemes based on the one or more manual sign features and the one or more eye markers;
producing a plurality of sign sequences from the phonemes;
parsing the sign sequences to produce grammatically parsed sign utterances;
translating the grammatically parsed sign utterances into grammatical representations in the target language; and
generating output utterances in the target language based upon the grammatical representations.
2. The method of claim 1 , wherein confidence values are produced for each generated output utterance.
3. The method of claim 1 , wherein producing the phonemes from the motion capture data includes producing a confidence value for each produced phoneme .
4. The method of claim 1 , wherein producing the phonemes includes producing a plurality of segments as time intervals matching the phonemes, where these intervals of the plurality of segments may overlap.
5. The method of claim 4 , wherein producing the phonemes and their intervals includes determining of a set of possible succeeding phonemes for each phoneme.
6. The method of claim 1 , wherein producing the sign sequences from the phonemes includes matching potential paths in a graph of phonemes to each sign in each sign sequence.
7. The method of claim 1 , wherein producing the grammatically parsed sign utterances includes producing a grammatical context and using the grammatical context of previous utterances.
8. The method of claim 1 , wherein producing the grammatically parsed sign utterances includes producing a confidence value based on the confidences of the signs, the confidence of the parsing and confidence of the parse matching a grammatical context for each parse of each sign sequence.
9. The method of claim 1 , further comprising
generating a plurality of output utterances in the target language based upon the plurality of sign sequences;
displaying the plurality output utterances to the signing user; and
receiving an indication from the signing user selecting one of the plurality of displayed output utterances as a correct translation.
10. The method of claim 1 , further comprising detecting an end of a sign language utterance before parsing the sign sequence to produce a grammatically parsed sign utterance.
11. The method of claim 1 , wherein the motion capture data includes data captured using marked gloves used by the signing user.
12. The method of claim 1 , wherein user specific parameter data is used for one of:
producing the phonemes;
producing the plurality of sign sequences;
parsing the sign sequences; and
translating the grammatically parsed sign utterances.
13. The method of claim 1 , further comprising:
in response to detecting that the signing user is using fingerspelling based on the phonemes, performing operations comprising:
generating fingerspelling phonemes based on the phonemes;
translating the fingerspelling phonemes to translated letters in the target language;
generating an output to the signing user showing the translated letters to the signing user; and
receiving an input from the signing user indicating a correctness of the translated letters.
14. A system configured to translate sign language utterances into a target language, comprising:
an input interface configured to receive motion capture data from a signing user making signs in the sign language;
a memory; and
a processor in communication with the input interface and the memory, the processor being configured to:
determine one or more manual sign features in parallel with one or more eye markers based on an independent temporal segmentation of the motion capture data for the manual sign features and the one or more eye markers;
produce phonemes based on the one or more manual sign features and the one or more eye markers;
produce a plurality of sign sequences from the phonemes;
parse the sign sequences to produce grammatically parsed sign utterances;
translate the grammatically parsed sign utterances into a grammatical representation in the target language; and
generate output utterances in the target language based upon the grammatical representation.
15. The system of claim 14 , wherein confidence values are produced for each generated output utterance.
16. The system of claim 14 , wherein producing the phonemes from motion capture data includes producing a confidence value for each produced phoneme.
17. The system of claim 14 , wherein producing the phonemes includes producing a plurality of segments as time intervals matching the phonemes, where the time intervals of the plurality of segments may overlap.
18. The system of claim 17 , wherein producing the phonemes and their intervals includes determining of a set of possible succeeding phonemes for each phoneme.
19. The system of claim 14 , wherein producing the sign sequences from the phonemes includes matching potential paths in a graph of phonemes to each sign in each sign sequence.
20. The system of claim 14 , wherein producing the grammatically parsed sign utterances includes producing a grammatical context and using the grammatical context of previous utterances.
21. The system of claim 14 , wherein producing the grammatically parsed sign utterances includes producing a confidence value based on the confidences of the signs, the confidence of the parsing and confidence of the parse matching a grammatical context for each parse of each sign sequence.
22. The system of claim 14 , wherein the processor is further configured to:
generate a plurality of output utterances in the target language based upon the plurality of sign sequences;
display the plurality output utterances to the user; and
receive an indication from the signing user selecting one of the plurality of displayed output utterances as a correct translation.
23. The system of claim 14 , wherein the processor is further configured to detect an end of a sign language utterance before parsing the sign sequence to produce a grammatically parsed sign utterance.
24. The system of claim 14 , wherein the motion capture data includes data captured using marked gloves used by the signing user.
25. The system of claim 14 , wherein user specific parameter data is used for one of:
producing the phonemes;
producing the plurality of sign sequences;
parsing the sign sequences; and
translating the grammatically parsed sign utterances.
26. The system of claim 14 , wherein the processor is further configured to:
in response to detecting that the signing user is using fingerspelling based on the phonemes, performing operations comprising:
generating fingerspelling phonemes based on the phonemes detecting that the signing user is using fingerspelling, after producing phonemes;
translating the fingerspelling phonemes to letters in the target language;
generating an output to the signing user showing translated letters to the signing user; and
receiving an input from the signing user indicating a correctness of the translated letters.
27. The system of claim 14 , further comprising sensors producing the motion capture data.
28. The system of claim 14 , further comprising marked gloves used by the signing.
29. The system of claim 14 , wherein the input interface further receives communication input from a second user to facilitate a conversation between the signing user and the second user.