Systems and method for action recognition using micro-doppler signatures and recurrent neural networks
The present disclosure may be embodied as systems and methods for action recognition developed using a multimodal dataset that incorporates both visual data, which facilitates the accurate tracking of movement, and active acoustic data, which captures the micro-Doppler modulations induced by the motion. The dataset includes twenty-one actions and focuses on examples of orientational symmetry that a single active ultrasound sensor should have the most difficulty discriminating. The combined results from three independent ultrasound sensors are encouraging, and provide a foundation to explore the use of data from multiple viewpoints to resolve the orientational ambiguity in action recognition. In various embodiments, recurrent neural networks using long short-term memory (LSTM) or hidden Markov models (HMMs) are disclosed for use in action recognition, for example, human action recognition, from micro-Doppler signatures.
1. A system for action recognition of sensed motion, comprising:
a first micro-Doppler sensor configured to produce a spectrogram of a sensed motion;
a splitter configured to receive the spectrogram from the first micro-Doppler sensor and segment the spectrogram into a sequence of spectrogram slices, each spectrogram slice of the sequence of spectrogram slices having a pre-determined time period;
an artificial neural network (“ANN”) for classifying sensed motion, the ANN comprising a recognition layer configured to receive the sequence of spectrogram slices from the splitter as an input vector.
2. The system of claim 1 , wherein the ANN is a recurrent neural network (“RNN”).
3. The system of claim 2 , wherein the recognition layer comprises a plurality of hidden Markov models (“HMM”), each HMM corresponding to a hidden sequence of skeletal poses and a visible sequence of spectrogram slice prototypes.
4. The system of claim 3 , wherein the recognition layer further comprises a Viterbi module configured to receive a plurality of spectrogram slice prototypes and to find a most likely sequence of skeletal pose prototypes.
5. The system of claim 2 , wherein the recognition layer comprises one or more long short-term memory layers.
6. The system of claim 1 , wherein the ANN is a convolutional deep belief network (“CDBN”).
7. The system of claim 1 , wherein the micro-Doppler sensor is an ultrasound sensor.
8. The system of claim 1 , further comprising one or more additional micro-Doppler sensors, each additional micro-Doppler sensor configured to produce a spectrogram of the sensed motion.
9. The system of claim 8 , wherein the splitter is further configured to concatenate the spectrograms from the first micro-Doppler sensor and the one or more additional micro-Doppler sensors; and wherein the recognition layer is configured to receive the concatenated spectrograms as the input vector.
10. The system of claim 8 , further comprising one or more additional ANNs, each ANN corresponding to an additional micro-Doppler sensor of the one or more additional micro-Doppler sensors and having a recognition layer configured to receive a plurality of spectrogram slices as an input vector.
11. The system of claim 1 , further comprising a motion sensor configured to produce RGB-D data of a sensed motion, and where the input vector further comprises a sequence of RGB-D data.
12. A method for action recognition of sensed motion, comprising:
capturing a spectrogram of a sensed motion using a micro-Doppler sensor;
segmenting the captured spectrogram into a sequence of spectrogram slices, each spectrogram slice of the sequence of spectrogram slices having a pre-determined time period; and
classifying the sequence of spectrogram slices as an action using an artificial neural network (“ANN”) having a recognition layer configured to receive the sequence of spectrogram slices as an input vector.
13. The method of claim 12 , further comprising translating each spectrogram slice of the sequence of spectrogram slices into a spectrogram slice prototype based on the Euclidean distance between each spectrogram slice and each spectrogram slice prototype.
14. The method of claim 13 , wherein the recognition layer comprises a plurality of hidden Markov models (“HMM”), each HMM corresponding to a hidden sequence of skeletal poses and a visible sequence of spectrogram slice prototypes; and classifying the sequence of spectrogram slices further comprises:
computing, using each HMM, a most likely sequence of hidden skeletal poses;
computing a log-likelihood of each of the most likely sequences of hidden sequence of skeletal poses and the visible sequence of spectrogram slice prototypes; and
selecting the action based on the computed log-likelihoods.
15. The method of claim 14 , wherein a Viterbi algorithm is used to compute a maximum a posteriori (MAP) estimate.
16. The method of claim 12 , further comprising capturing additional spectrograms of the sensed motion using additional micro-Doppler sensors.
17. The method of claim 16 , further comprising concatenating the spectrogram and the additional spectrograms.