Small unmanned aerial systems detection and classification using multi-modal deep neural networks
View Patent ↗Provided is a detection and classification system and method for small unmanned aircraft systems (sUAS). The system and method detect and classify multiple simultaneous heterogeneous RC transmitters/sUAS downlinks from the RF signature using Object Detection Deep Convolutional Neural Networks (DCNNs). The method further utilizes not only passive RF, but may also utilize Electro Optic/Infrared (EO/IR), radar and acoustic sensors as well, with a fusion of the individual sensor classifications. Detection and classification with Identification Friend or Foe (IFF) of individual sUAS in a swarm, multi-modal approach for high confidence classification, decision, and implementation on a low C-SWaP (cost, size, weight and power) NVIDIA Jetson TX2 embedded AI platform is achieved.
1 . A small unmanned aerial systems (SUAS) detection and classification system comprising:
a radio frequency (RF) sensor configured to monitor radio frequencies in an environment in which at least one sUAS is being operated and determine one or more RF spectrograms based on the monitored radio frequencies;
an optical sensor configured to capture at least one of electro-optic information or infrared (IR) information about the at least one SUAS;
a radar sensor configured to measure radar information and determine one or more radar spectrograms from the measured radar information for the environment in which the at least one sUAS is being operated;
a sound sensor configured to record acoustical information in the environment in which the at least one SUAS is being operated and generate one or more acoustic spectrograms based on the recorded acoustical information; and
a plurality of deep convolutional neural networks (DCNNs) wherein each of the plurality of DCNNs is respectively coupled with the RF sensor, the optical sensor, the radar sensor, and the sound sensor, the DCNNs configured to perform deep learning to identify and/or classify the at least one SUAS based on one or more of the one or more RF spectrograms, the electro-optic information or infrared (IR) information, the one or more radar spectrograms, and the one or more acoustic spectrograms;
wherein at least one DCNN of the plurality of DCNNs is further configured to receive trained weights input from a training processor for determining identification and/or classification of the at least one SUAS; and
wherein the at least one DCNN of the plurality of DCNNs comprises a Siamese DCNN (SDCNN) configured to train on nonlinearities of a plurality of radio transmitters.
2 . The SUAS detection and classification system of claim 1 , wherein the system is configured to:
collect at least one frequency hop sequence of an RF signal corresponding to the at least one SUAS.
3 . The SUAS detection and classification system of claim 2 , wherein the system is configured to:
train at least one long short-term memory (LSTM) recurrent neural network (RNN) based on the collected at least one frequency hop sequence; and
predict subsequent transmission of next hop frequencies of the at least one frequency hop sequence corresponding to the at least one SUAS using the LSTM RNN trained based on the collected at least one frequency hop sequence.
4 . The SUAS detection and classification system of claim 1 , further comprising:
an ultra wideband (UWB) omnidirectional antenna coupled to the RF sensor.
5 . The SUAS detection and classification system of claim 1 , wherein the SDCNN comprises:
a first Siamese DCNN configured to receive a first input and output a first weighted data;
a second Siamese DCNN configured to receive a second input and output a second weighted data, wherein the first and second Siamese DCNNs are configured to share the input trained weights;
a distance module configured to compute a Euclidean distance between the output first and second weighted data; and
a hinge embedding loss (HEL) computation module configured to measure the degree of similarity of the first and second inputs from the Euclidean distance and determine a gradient of the loss with respect to DCNN parameters of the SDCNN.
6 . The SUAS detection and classification system of claim 5 , wherein the SDCNN is further configured to propagate the gradient of the loss backwards using a back propagation algorithm to update the DCNN parameters of the SDCNN such that the HEL is a first value when the first and second inputs originate from a same RF transmitter for a particular SUAS and a second value when the first and second inputs respectively originate from different RF transmitters for two respective particular sUASs, wherein the first value is less than the second value.
7 . A method for small unmanned aerial systems (SUAS) detection and classification comprising:
monitoring radio frequencies using a radio frequency (RF) sensor in an environment in which at least one SUAS is being operated and determine one or more RF spectrograms based on the monitored radio frequencies;
capturing at least one of electro-optic information or infrared (IR) information about the at least one SUAS;
measuring radar information and determine one or more radar spectrograms from the measured radar information for the environment in which the at least one SUAS is being operated;
recording acoustical information in the environment in which the at least one SUAS is being operated and generate one or more acoustic spectrograms based on the recorded acoustical information; and
identifying and/or classifying the at least one sUAS using a plurality of deep convolutional neural networks (DCNNs) wherein each of the plurality of DCNNs is respectively coupled with the RF sensor, the optical sensor, the radar sensor, and the sound sensor, the DCNNs configured to perform deep learning for identifying and/or classifying the at least one SUAS based on one or more of the one or more RF spectrograms, the electro-optic information or infrared (IR) information, the one or more radar spectrograms, and the one or more acoustic spectrograms;
wherein at least one DCNN of the plurality of DCNNs is further configured to receive trained weights input from a training processor for determining identification and/or classification of the at least one SUAS; and
wherein the at least one DCNN of the plurality of DCNNs comprises a Siamese DCNN (SDCNN) configured to train on nonlinearities of a plurality of radio transmitters.
8 . The method for sUAS detection and classification of claim 7 , further comprising:
collecting at least one frequency hop sequence of an RF signal corresponding to the at least one SUAS.
9 . The method for SUAS detection and classification of claim 8 , further comprising:
training at least one long short-term memory (LSTM) recurrent neural network (RNN) based on the collected at least one frequency hop sequence; and
predicting subsequent transmission of next hop frequencies of the at least one frequency hop sequence corresponding to the at least one SUAS using the LSTM RNN trained based on the collected at least one frequency hop sequence.
10 . The method for SUAS detection and classification of claim 7 , wherein the SDCNN comprises:
a hinge embedding loss (HEL) computation module configured to measure the degree of similarity of the first and second inputs from the Euclidean distance and determine a gradient of the loss with respect to DCNN parameters of the SDCNN,
a first Siamese DCNN configured to receive a first input and output a first weighted data;
a second Siamese DCNN configured to receive a second input and output a second weighted data, wherein the first and second Siamese DCNNs are configured to share the input trained weights;
a distance module configured to compute a Euclidean distance between the output first and second weighted data; and
a hinge embedding loss (HEL) computation module configured to measure the degree of similarity of the first and second inputs from the Euclidean distance and determine a gradient of the loss with respect to DCNN parameters of the SDCNN.
11 . The method for SUAS detection and classification of claim 10 , wherein the SDCNN is further configured to propagate the gradient of the loss backwards using a back propagation algorithm to update the DCNN parameters of the SDCNN such that the HEL is a first value when the first and second inputs originate from a same RF transmitter for a particular sUAS and a second value when the first and second inputs respectively originate from different RF transmitters for two respective particular sUASs, wherein the first value is less than the second value.