Realtime AI sign language recognition
A real time sign language recognition method that allows Deaf and Hard of Hearing individuals to sign into any apparatus with a camera to extract target information (such as a translation in a target language) is proposed.
1 . A method, comprising:
capturing a sequence of images;
for an image within the sequence of images, detecting pose information by:
applying a first pose network to detect a body pose configuration in the image;
applying a second pose network to detect a hand pose configuration in the image;
generating a feature vector including the body pose configuration and the hand pose configuration; and
converting the feature vector into a flattened feature vector having a predefined size;
generating a feature queue by collecting the flattened feature vector for the image of the sequence of images; and
converting the feature queue into a target language output.
2 . The method of claim 1 , wherein converting the feature queue comprises:
splitting the feature queue into individual regions; and
processing the individual regions into a sign language string.
3 . The method of claim 2 , wherein processing the individual regions comprises determining whether the individual regions are one of a pre-recorded sentence or an individual sign in one or more databases.
4 . The method of claim 2 , wherein processing the individual regions comprises applying a binary classifier to determine whether one or more of the individual regions is fingerspelled.
5 . The method of claim 2 , wherein processing the individual regions comprises:
comparing the individual regions to signs in one or more databases to generate comparison results; and
choosing a sign based on a K Nearest Neighbor function or a Dynamic Time Warping function applied to the comparison results.
6 . The method of claim 1 , wherein converting the feature queue comprises applying a Convolutional Neural Network (CNN) configured to output one or more flag values associated with an intrasign region, an intersign region, or a non-signing region, and wherein the one or more flag values correspond to an individual sign.
7 . A method, comprising:
detecting hand pose information from a sequence of image frames;
converting the hand pose information into a flattened feature vector;
normalizing the flattened feature vector into a resultant feature vector;
applying a convolutional neural network (CNN) to split the resultant feature vector into a plurality of individual regions by highlighting sign transition periods;
applying the CNN to output a respective flag corresponding to each individual region of the plurality of individual regions, wherein the respective flag indicates whether the corresponding individual region is an intrasign region, intersign region, or a nonsigning region, and wherein the intrasign and intersign regions correspond to signing regions of the plurality of individual regions;
processing the signing regions of the plurality of individual regions into a sign language string based on language information in one or more databases; and
translating the sign language string into a target language output.
8 . The method of claim 7 , wherein normalizing the flattened feature vector comprises:
setting head coordinates to be (0,0) in a pose and shoulders to be an average of 1 unit via a first affine transform; and
setting mean coordinates of one or more hands to be (0,0,0), wherein a standard deviation in the mean coordinates is an average of 1 unit via a second affine transform.
9 . The method of claim 7 , wherein processing the signing regions comprises determining whether the signing regions are one of a pre-recorded sentence or an individual sign in the one or more databases by applying a K Nearest Neighbor function or a Dynamic Time Warping function on the individual regions.
10 . The method of claim 7 , wherein processing the signing regions comprises applying a binary classifier to determine whether one or more of the signing regions is fingerspelled.
11 . The method of claim 7 , wherein processing the signing regions comprises:
comparing one or more of the signing regions to signs in the one or more databases to generate comparison results; and
choosing a sign based on a K Nearest Neighbor function or a Dynamic Time Warping function applied to the comparison results.
12 . The method of claim 7 , further comprising outputting a plurality of selectable translations associated with the target language output.
13 . The method of claim 7 , wherein normalizing the flattened feature vector comprises setting coordinates corresponding to a user head to be at an origin and coordinates corresponding to user shoulders to be at a unit distance from the origin.
14 . A device, comprising:
a camera configured to capture a sequence of image frames;
a computing device coupled to the camera, the computing device comprising:
a processor; and
a memory, wherein the memory contains instructions stored thereon that when executed by the processor cause the processor to:
detect hand pose information from the sequence of image frames;
convert the hand pose information into a flattened feature vector;
normalize the flattened feature vector into a resultant feature vector;
apply a convolutional neural network (CNN) to split the resultant feature vector into a plurality of individual regions by highlighting sign transition periods;
apply the CNN to output a respective flag corresponding to each individual region of the plurality of individual regions, wherein the respective flag indicates whether the corresponding individual region is an intrasign region, intersign region, or a nonsigning region, and wherein the intrasign and intersign regions correspond to signing regions of the plurality of individual regions;
process the signing regions of the plurality of individual regions into a sign language string based on language information in one or more databases; and
translate the sign language string into a target language output.
15 . The device of claim 14 , wherein to process the signing regions, the processor is configured to determine whether the signing regions are one of a pre-recorded sentence or an individual sign in the one or more databases by applying a K Nearest Neighbor function or a Dynamic Time Warping function on the individual regions.
16 . The device of claim 14 , wherein to process the signing regions, the processor is configured to apply a binary classifier to determine whether one or more of the signing regions is fingerspelled.
17 . The device of claim 14 , wherein to process the signing regions, the processor is configured to:
compare the signing regions to signs in the one or more databases to generate comparison results; and
choose a sign based on a K Nearest Neighbor function or a Dynamic Time Warping function applied to the comparison results.
18 . The device of claim 14 , wherein the processor is further configured to output a plurality of selectable translations associated with the target language output.
19 . The device of claim 14 , wherein the processor is further configured to use an affine transformation to normalize the flattened feature vector.
20 . The device of claim 14 , wherein to normalize the flattened feature vector, the processor is configured to set coordinates corresponding to a user head to be at an origin and coordinates corresponding to user shoulders to be at a unit distance from the origin.