COMPUTER VISION BASED SIGN LANGUAGE INTERPRETER
A system and method for translating sign language utterances into a target language, including: receiving motion capture data; producing phonemes/sign fragments from the received motion capture data; producing a plurality of sign sequences from the phonemes/sign fragments; parsing these sign sequences to produce grammatically parsed sign utterances; translating the grammatically parsed sign utterances into grammatical representations in the target language; and generating output utterances in the target language based upon the grammatical representations.
1 . A computer-implemented method comprising:
capturing motion capture data of a sign user;
extracting a set of non-manual markers from the motion capture data, the non-manual markers comprising:
a set of lip features selected from a group consisting of a flattened lip feature, a retracted lip feature, or a protruding lip feature;
a set of spoken word articulations selected from a group consisting of a vowel articulation or a consonant articulation; and
a set of head movements; and
generating a set of phonemes with an associated set of confidence values using the set of non-manual markers;
producing a set of sign utterances based on the set of phonemes; and
translating the set of sign utterances into a target language.
2 . The computer-implemented method of claim 1 , wherein extracting the set of non-manual markers comprises extracting a set of upper face features selected from a group including a raised eyebrow movement, a lowered eyebrow movement, or a knitted eyebrow movement.
3 . The computer-implemented method of claim 1 , wherein generating the set of phonemes comprises producing a plurality of segments as a set of time intervals matching the set of phonemes.
4 . The computer-implemented method of claim 1 , wherein generating the set of phonemes further comprises determining a set of possible succeeding phonemes for each phoneme.
5 . The computer-implemented method of claim 1 , wherein producing the set of sign utterances comprises producing a grammatical context and using a previous grammatical context of previous sign utterances.
6 . The computer-implemented method of claim 1 , wherein generating the set of phonemes further comprises using user specific parameter data.
7 . The computer-implemented method of claim 1 , wherein producing the set of sign utterances further comprises using user specific parameter data.
8 . A system comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
capturing motion capture data of a sign user;
extracting a set of non-manual markers from the motion capture data, the non-manual markers comprising:
a set of lip features selected from a group consisting of a flattened lip feature, a retracted lip feature, or a protruding lip feature;
a set of spoken word articulations selected from a group consisting of a vowel articulation or a consonant articulation; and
a set of head movements; and
generating a set of phonemes with an associated set of confidence values using the set of non-manual markers;
producing a set of sign utterances based on the set of phonemes; and
translating the set of sign utterances into a target language.
9 . The system of claim 8 , wherein extracting the set of non-manual markers comprises extracting a set of upper face features selected from a group including a raised eyebrow movement, a lowered eyebrow movement, or a knitted eyebrow movement.
10 . The system of claim 8 , wherein generating the set of phonemes comprises producing a plurality of segments as a set of time intervals matching the set of phonemes.
11 . The system of claim 8 , wherein generating the set of phonemes further comprises determining a set of possible succeeding phonemes for each phoneme.
12 . The system of claim 8 , wherein producing the set of sign utterances comprises producing a grammatical context and using a previous grammatical context of previous sign utterances.
13 . The system of claim 8 , wherein generating the set of phonemes further comprises using user specific parameter data.
14 . The system of claim 8 , wherein producing the set of sign utterances further comprises using user specific parameter data.
15 . A non-transitory machine-readable medium including instructions that, when executed by a system, cause the system to perform operations comprising:
capturing motion capture data of a sign user;
extracting a set of non-manual markers from the motion capture data, the non-manual markers comprising:
a set of lip features selected from a group consisting of a flattened lip feature, a retracted lip feature, or a protruding lip feature;
a set of spoken word articulations selected from a group consisting of a vowel articulation or a consonant articulation; and
a set of head movements; and
generating a set of phonemes with an associated set of confidence values using the set of non-manual markers;
producing a set of sign utterances based on the set of phonemes; and
translating the set of sign utterances into a target language.
16 . The non-transitory machine-readable medium of claim 15 , wherein extracting the set of non-manual markers comprises extracting a set of upper face features selected from a group including a raised eyebrow movement, a lowered eyebrow movement, or a knitted eyebrow movement.
17 . The non-transitory machine-readable medium of claim 15 , wherein generating the set of phonemes comprises producing a plurality of segments as a set of time intervals matching the set of phonemes.
18 . The non-transitory machine-readable medium of claim 15 , wherein generating the set of phonemes further comprises determining a set of possible succeeding phonemes for each phoneme.
19 . The non-transitory machine-readable medium of claim 15 , wherein producing the set of sign utterances comprises producing a grammatical context and using a previous grammatical context of previous sign utterances.
20 . The non-transitory machine-readable medium of claim 15 , wherein generating the set of phonemes further comprises using user specific parameter data.
21 . The non-transitory machine-readable medium of claim 15 , wherein producing the set of sign utterances further comprises using user specific parameter data.