Computer vision based sign language interpreter
A system and method for translating sign language utterances into a target language, including: receiving motion capture data; producing phonemes/sign fragments from the received motion capture data; producing a plurality of sign sequences from the phonemes/sign fragments; parsing these sign sequences to produce grammatically parsed sign utterances; translating the grammatically parsed sign utterances into grammatical representations in the target language; and generating output utterances in the target language based upon the grammatical representations.
1. A computer-implemented method comprising:
receiving motion capture data of a sign language from a user;
extracting a plurality of features from the motion capture data in parallel for two or more groups of features, wherein each group of features includes an independent temporal segmentation, wherein each group of features includes features extracted from the motion capture data from a plurality of channels, and wherein the plurality of channels includes a respective channel for each hand of the user, a body of the user, and a face of the user;
aggregating the groups of features into an aggregate segmentation;
determining one or more sign patterns in the aggregate segmentation, each sign pattern including a weight distribution defining an importance of each channel of the plurality of channels;
recognizing one or more signs using one or more respective sign patterns of the one or more sign patterns;
producing a plurality of potential sign sequences using the one or more signs;
parsing the potential sign sequences to produce parsed sign utterances; and
generating output utterances in a target language using the parsed sign utterances.
2. The computer-implemented method of claim 1 , further comprising:
displaying multiple candidate output utterances to the user; and
receiving a selection from the user of a correct output utterance from the multiple candidate output utterances.
3. The computer-implemented method of claim 1 , wherein extracting the plurality of features comprises extracting sign features from one or more hands and one or more arms of the user, and eye gaze features from one or more eyes of the user.
4. The computer-implemented method of claim 1 , wherein recognizing one or more signs comprises generating one or more phonemes and associated confidence values indicating confidence in a phoneme.
5. The computer-implemented method of claim 4 , wherein producing potential sign sequences comprises matching paths through a graph of the phonemes.
6. The computer-implemented method of claim 1 , wherein parsing comprises using a grammatical context from previous utterances in a same conversation session to determine proper parsing of a current utterance.
7. The computer-implemented method of claim 1 , further comprising applying user parameters at one or more of operations of: extracting the plurality of features, recognizing the one or more signs, producing the plurality of potential sign sequences, parsing the potential sign sequences, and generating output utterances in the target language.
8. A machine, comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving motion capture data of a sign language from a user;
extracting a plurality of features from the motion capture data in parallel for two or more groups of features, wherein each group of features includes an independent temporal segmentation, wherein each group of features includes features extracted from the motion capture data from a plurality of channels, and wherein the plurality of channels includes a respective channel for each hand of the user, a body of the user, and a face of the user;
aggregating the groups of features into an aggregate segmentation;
determining one or more sign patterns in the aggregate segmentation, each sign pattern including a weight distribution defining an importance of each channel of the plurality of channels;
recognizing one or more signs using one or more respective sign patterns of the one or more sign patterns;
producing a plurality of potential sign sequences using the one or more signs;
parsing the potential sign sequences to produce parsed sign utterances; and
generating output utterances in a target language using the parsed sign utterances.
9. The machine of claim 8 , wherein the operations further comprise:
displaying multiple candidate output utterances to the user; and
receiving a selection from the user of a correct output utterance from the multiple candidate output utterances.
10. The machine of claim 8 , wherein extracting the plurality of features comprises extracting manual sign features from one or more hands and one or more arms of the user, and eye gaze features from one or more eyes of the user.
11. The machine of claim 8 , wherein recognizing one or more signs comprises generating one or more phonemes and associated confidence values indicating confidence in a phoneme.
12. The machine of claim 11 , wherein producing potential sign sequences comprises matching paths through a graph of the phonemes.
13. The machine of claim 8 , wherein parsing comprises using a grammatical context from previous utterances in a same conversation session to determine proper parsing of a current utterance.
14. The machine of claim 8 , wherein the operations further comprise applying user parameters at one or more of the operations of: extracting the plurality of features, recognizing the one or more signs, producing the plurality of potential sign sequences, parsing the potential sign sequences, and generating output utterances in the target language.
15. A non-transitory machine-readable storage medium storing machine-executable instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving motion capture data of a sign language from a user;
extracting a plurality of features from the motion capture data in parallel for two or more groups of features, wherein each group of features includes an independent temporal segmentation, wherein each group of features includes features extracted from the motion capture data from a plurality of channels, and wherein the plurality of channels includes a respective channel for each hand of the user, a body of the user, and a face of the user;
aggregating the groups of features into an aggregate segmentation;
determining one or more sign patterns in the aggregate segmentation, each sign pattern including a weight distribution defining an importance of each channel of the plurality of channels;
recognizing one or more signs using one or more respective sign patterns of the one or more sign patterns;
producing a plurality of potential sign sequences using the one or more signs;
parsing the potential sign sequences to produce parsed sign utterances; and
generating output utterances in a target language using the parsed sign utterances.
16. The non-transitory machine-readable storage medium of claim 15 , wherein extracting the plurality of features comprises extracting manual sign features from one or more hands and one or more arms of the user, and eye gaze features from one or more eyes of the user.
17. The non-transitory machine-readable storage medium of claim 15 , wherein recognizing one or more signs comprises generating one or more phonemes and associated confidence values indicating confidence in a phoneme.
18. The non-transitory machine-readable storage medium of claim 17 , wherein producing potential sign sequences comprises matching paths through a graph of the phonemes.
19. The non-transitory machine-readable storage medium of claim 15 , wherein parsing comprises using a grammatical context from previous utterances in a same conversation session to determine proper parsing of a current utterance.
20. The non-transitory machine-readable storage medium of claim 15 , wherein the operations further comprise applying user parameters at one or more of the operations of: extracting the plurality of features, recognizing the one or more signs, producing the plurality of potential sign sequences, parsing the potential sign sequences, and generating output utterances in the target language.