Automated gesture identification using neural networks
Disclosed are methods, apparatus and systems for gesture recognition based on neural network processing. One exemplary method for identifying a gesture communicated by a subject includes receiving a plurality of images associated with the gesture, providing the plurality of images to a first 3-dimensional convolutional neural network (3D CNN) and a second 3D CNN, where the first 3D CNN is operable to produce motion information, where the second 3D CNN is operable to produce pose and color information, and where the first 3D CNN is operable to implement an optical flow algorithm to detect the gesture, fusing the motion information and the pose and color information to produce an identification of the gesture, and determining whether the identification corresponds to a singular gesture across the plurality of images using a recurrent neural network that comprises one or more long short-term memory units.
1. A device for processing images associated with a gesture, comprising:
at least one camera; and
at least one processor configured to implement:
one or more three-dimensional convolution neural networks (3D CNNs), each of the 3D CNNs comprising:
an input to receive a plurality of input images from the at least one camera, and
an output to provide recognition information produced by each of the 3D CNNs, and
at least one recurrent neural network (RNN) comprising:
an input to receive a second type of recognition information, and
an output that is coupled to the input of the at least one RNN to provide a feedback connection,
wherein the at least one processor is configured to:
receive a plurality of captured images at a pre-processing module, perform pose estimation on each of the plurality of captured images, and overlay pose estimation pixels onto the plurality of captured images to generate the plurality of input images for consumption by the one or more 3D CNNs, and
receive the recognition information produced by each of the one or more 3D CNNs at a fusion module, and aggregate the received recognition information to generate the second type of recognition information for consumption by the at least one RNN,
wherein each of the one or more 3D CNNs is operable to produce the recognition information comprising at least one characteristic associated with the gesture in each of the plurality of input images, and provide the recognition information to the fusion module, the at least one characteristic comprising a pose, a color or a gesture type, and
wherein the at least one RNN is operable to determine whether the recognition information produced by the one or more 3D CNNs corresponds to a singular gesture across the plurality of input images.
2. The device of claim 1 , wherein a first of the one or more 3D CNNs is operable to use an optical flow algorithm to detect the gesture.
3. The device of claim 2 , wherein the optical flow algorithm comprises sharpening, line, edge, corner and shape enhancements.
4. The device of claim 1 , wherein the pose estimation identifies a body, fingers and face of a subject performing the gesture.
5. The device of claim 1 , wherein the one or more 3D CNNs and the at least one RNN are implemented on the at least one processor.
6. The device of claim 1 , wherein the at least one processor is configured to:
process the plurality of captured images to extract pixels corresponding to a subject, a foreground and a background in each of the plurality of captured images;
perform spatial filtering, upon determining that the background or the foreground includes no information that pertains to the gesture being identified, to remove the pixels corresponding to the background or the foreground.
7. The device of claim 1 , wherein a first 3D CNN of the one or more 3D CNNs is operable to use an optical flow algorithm to detect the gesture, and wherein the at least one characteristic identified by the first 3D CNN comprises the gesture type.
8. The device of claim 7 , wherein a second 3D CNN of the one or more 3D CNNs is operable to process the pose estimation pixels to associate a pose with the gesture, and wherein the at least one characteristic identified by the second 3D CNN comprises the pose.
9. The device of claim 8 , wherein a third 3D CNN of the one or more 3D CNNs is operable to process an expert analysis of the gesture, and wherein the at least one characteristic identified by the third 3D CNN comprises the gesture type.
10. The device of claim 1 , wherein the at least one RNN comprises one or more long short-term memory (LTSM) units.
11. The device of claim 1 , wherein the fusion module is operable to concurrently process at least one output node of each of the one or more 3D CNNs.
12. The device of claim 1 , wherein the at least one camera comprises at least one of a stereo camera, a camera augmented with a depth sensor, a multi-aperture camera, or a right camera and a left camera.
13. The device of claim 1 , comprising:
a left microphone and a right microphone configured to capture environmental and contextual cues that are different from the plurality of captured images.
14. The device of claim 1 , comprising a front display.
15. The device of claim 14 , comprising a rear display that is a touchscreen display.
16. The device of claim 1 , wherein the at least one processor comprises at least one graphics processing unit (GPU).