Smart-device-based radar system performing gesture recognition using a space time neural network
Techniques and apparatuses are described that implement a smart-device-based radar system capable of performing gesture recognition using a space time neural network. The space time neural network employs machine learning to recognize a user's gesture based on complex radar data. The space time neural network is implemented using a multi-stage machine-learning architecture, which enables the radar system to conserve power and recognize the user's gesture in real time (e.g., as the gesture is performed). The space time neural network is also adaptable and can be expanded to recognize multiple types of gestures, such as a swipe gesture and a reach gesture, without significantly increasing size, computational requirements, or latency.
1. A method performed by a radar system, the method comprising:
transmitting a radar transmit signal using an antenna array of the radar system;
receiving a radar receive signal using the antenna array, the radar receive signal comprising a version of the radar transmit signal that is reflected by at least one user;
generating complex radar data based on the radar receive signal;
providing the complex radar data to a space time neural network of the radar system, the space time neural network comprising a space-recurrent network and a time-recurrent network; and
analyzing the complex radar data using the space time neural network to recognize a gesture performed by the at least one user, the analyzing of the complex radar data comprising:
generating channel-Doppler data for each range bin associated with the complex radar data by separately processing portions of the complex radar data associated with different range bins using the space-recurrent network;
generating feature data associated with the gesture by analyzing the channel-Doppler data across the different range bins using the space-recurrent network; and
forming a prediction regarding the gesture by analyzing the feature data associated with at least two different time frames using the time-recurrent network.
2. The method of claim 1 , wherein:
the analyzing of the complex radar data comprises analyzing both magnitude and phase information of the complex radar data using machine-learning techniques to recognize the gesture.
3. The method of claim 1 , further comprising:
storing the feature data within a circular buffer; and
accessing, by the time-recurrent network, the feature data stored within the circular buffer.
4. The method of claim 1 , wherein the separately processing portions of the complex radar data comprises separately processing portions of the complex radar data associated with the different range bins using a non-linear activation function to generate the channel-Doppler data for each range bin.
5. The method of claim 1 , wherein the complex radar data comprises at least one of the following:
a complex range-Doppler map;
complex interferometry data;
multiple digital beat signals associated with the radar receive signal; or
frequency-domain representations of the multiple digital beat signals.
6. The method of claim 1 , wherein the gesture comprises at least one of the following:
a swipe gesture;
a reach gesture;
a knob-turning gesture; or
a spindle-twisting gesture.
7. The method of claim 1 , wherein:
the receiving of the radar receive signal comprises receiving multiple versions of the radar receive signal using different antenna elements of the radar system; and
the generating of the complex radar data comprises generating digital beat signals using respective receive channels of the radar system, the respective receive channels connected to the different antenna elements, respectively.
8. An apparatus comprising:
a radar system comprising:
an antenna array;
a transceiver configured to:
transmit a radar transmit signal using the antenna array;
receive a radar receive signal using the antenna array, the radar receive signal comprising a version of the radar transmit signal that is reflected by at least one user; and
generate complex radar data based on the radar receive signal; and
a processor and computer-readable storage medium configured to implement a space time neural network configured to:
accept the complex radar data;
generate channel-Doppler data for each range bin associated with the complex radar data by separately processing portions of the complex radar data associated with different range bins using a space-recurrent network of the space time neural network;
generate feature data associated with a gesture performed by the at least one user by analyzing the channel-Doppler data across the different range bins using the space-recurrent network; and
form a prediction regarding the gesture by analyzing the feature data associated with at least two different time frames using a time-recurrent network of the space time neural network.
9. The apparatus of claim 8 , wherein the apparatus comprises a smart device, the smart device comprising one of the following:
a smartphone;
a smart watch;
a smart speaker;
a smart thermostat;
a security camera;
a vehicle; or
a household appliance.
10. The apparatus of claim 8 , further comprising:
a circular buffer configured to store the feature data,
wherein the space time neural network is configured to form the prediction by accessing the feature data stored within the circular buffer using the time-recurrent network.
11. The apparatus of claim 8 , wherein:
the antenna array comprises multiple antenna elements; and
the transceiver is configured to:
receive multiple versions of the radar receive signal using the multiple antenna elements; and
generate the complex radar data based on the multiple versions of the radar receive signal.
12. The apparatus of claim 8 , wherein:
the space time neural network is configured to recognize the gesture based on the prediction; and
the gesture comprises at least one of the following:
a swipe gesture;
a reach gesture;
a knob-turning gesture; or
a spindle-twisting gesture.
13. A non-transitory computer-readable storage medium comprising computer-executable instructions that, responsive to execution by a processor, implement:
a space time neural network configured to:
accept complex radar data associated with a radar receive signal that is reflected by at least one object;
generate channel-Doppler data for each range bin associated with the complex radar data by separately processing portions of the complex radar data associated with different range bins using a space-recurrent network of the space time neural network;
generate feature data associated with a gesture performed using the object by analyzing the channel-Doppler data across the different range bins using the space-recurrent network of the space time neural network; and
form a prediction regarding the gesture by analyzing the feature data associated with at least two different time frames using a time-recurrent network of the space time neural network.
14. The computer-readable storage medium of claim 13 , wherein:
the space time neural network is further configured to recognize the gesture based on the prediction; and
the gesture comprises at least one of the following:
a swipe gesture;
a reach gesture;
a knob-turning gesture; or
a spindle-twisting gesture.
15. The computer-readable storage medium of claim 14 , wherein:
the at least one object comprises a user;
the space time neural network is configured to communicate the recognized gesture to a radar-based application and
the computer-executable instructions, responsive to execution by the processor, implement the radar-based application configured to display content to the user based on the recognized gesture.
16. The computer-readable storage medium of claim 15 , wherein the space time neural network is further configured to:
prompt the user to perform the gesture;
record the complex radar data as training data while the user performs the gesture; and
execute a training procedure to recognize a future gesture performed by the user.
17. The computer-readable storage medium of claim 13 , wherein:
the at least one object comprises a stylus; and
the gesture is performed by a user using the stylus.
18. The computer-readable storage medium of claim 13 , wherein:
the at least one object comprises an inanimate object and a user;
the space time neural network is configured to identify a potential collision between the inanimate object and the user based on the prediction; and
the computer-executable instructions, responsive to execution by the processor, implement a radar-based application configured to alert the user to the potential collision.
19. The computer-readable storage media medium of claim 13 , wherein:
the at least one object comprises a user;
the space time neural network is configured to measure a vital sign of the user; and
the computer-executable instructions, responsive to execution by the processor, implement a radar-based application configured to inform the user of the measured vital sign.
20. The computer-readable storage medium of claim 13 , wherein the computer-executable instructions, responsive to execution by the processor, implement a hardware-abstraction module configured to:
generate hardware-agnostic complex radar data based on the complex radar data; and
provide, to the space time neural network, the hardware-agnostic complex radar data as the complex radar data.