Explainable squeeze-excitation multi-scale fusion network system and method for aerial scene and coastal area recognition using remote sensing images
A method, and a system for automatic classification of land use and land cover includes capturing remote sensing (RS) images by a sensor on an unmanned aerial vehicle (UAV). The method includes transmitting the RS images by UAV communication circuitry. The method includes receiving the RS images by server communication circuitry and storing the RS images in memory. The method includes iteratively training, by a machine learning engine, a deep learning network that integrates an input layer, at least one residual block, inception blocks, bottleneck blocks and inverted bottleneck blocks, dense blocks, and at least one squeeze-excitation block. The method includes inputting the RS images, preventing a training gradient from vanishing, extracting features at multiple scales, balancing computational cost, and promoting reuse of the extracted features. The method includes determining, by a softmax layer, at least one class label and a respective confidence score.
1 . A system for automatic classification of land use and land cover (LULC), comprising:
an unmanned aerial vehicle (UAV) including
a sensor for capturing a plurality of remote sensing (RS) images, and
UAV communication circuitry configured to transmit the plurality of RS images;
a server computer configured with
server communication circuitry configured to receive the plurality of RS images,
a memory configured to store the plurality of RS images,
a machine learning engine configured to iteratively train a deep learning network,
wherein the machine learning engine integrates:
an input layer configured to input the plurality of RS images from the memory,
at least one residual block to prevent a training gradient from vanishing,
a plurality of inception blocks to extract features at multiple scales,
a plurality of bottleneck blocks and inverted bottleneck blocks to balance computational cost,
a plurality of dense blocks to promote reuse of the extracted features, and
a softmax layer that determines at least one class label and a respective confidence score for the at least one class label based on the extracted features.
2 . The system of claim 1 , wherein the machine learning engine configured to iteratively train the deep learning network further integrates at least one squeeze-excitation (SE) block to emphasize a region of a RS image.
3 . The system of claim 2 , wherein the machine learning engine configured to iteratively train the deep learning network further integrates
a plurality of the squeeze-excitation (SE) blocks, each said SE block passes input features through a global average pooling layer to perform a squeeze operation through which a scalar value is calculated for each channel, a resulting squeezed vector is reshaped to (1, 1,C), where C is a number of channels, and two dense layers are applied to the reshaped vector, wherein a first of the dense layers reduces the number of channels and a second of the dense layers restores the original number of channels.
4 . The system of claim 3 , wherein the machine learning engine is configured to iteratively train the deep learning network such that each inception block is embedded after a respective said SE block including a plurality of parallel layers, each said parallel layer having different kernel sizes of 1×1, 3×3, and 5×5.
5 . The system of claim 1 , wherein the machine learning engine is configured to iteratively train the deep learning network such that each bottleneck block includes a compression layer configured to reduce spatial dimensions of the extracted features.
6 . The system of claim 3 , wherein the machine learning engine configured to iteratively train the deep learning network further integrates
three bottleneck blocks, wherein each bottleneck block is integrated with a respective SE block, a first of the bottleneck blocks has a stride of 2 to reduce spatial dimensions, and the other two bottleneck blocks have a stride of 1 to enhance the extracted features.
7 . The system of claim 3 , wherein the machine learning engine configured to iteratively train the deep learning network such that each dense block is connected to a respective inverted bottleneck block,
wherein each layer in a dense block is connected directly to all subsequent layers, and
wherein each dense block comprises four layers, each consisting of a 1×1 convolutional layer, a 3×3 convolutional layer, a concatenation operation, and a respective squeeze and excitation block.
8 . The system of claim 1 , wherein the machine learning engine is further configured to apply a crossentropy loss function in order to train the deep learning network.
9 . The system of claim 1 , wherein the unmanned aerial vehicle (UAV) further includes
an on-board machine learning engine configured with the trained deep learning network for real-time classification of LULC,
wherein the sensor is further configured to capture, every predetermined time period, a remote sensing (RS) image, and
wherein the on-board machine learning engine is configured to classify the RS image at the predetermined time period.
10 . The system of claim 9 , wherein the UAV communication circuitry is further configured to transmit a plurality of RS images captured over a plurality of the predetermined time periods, together with a respective class and confidence score.
11 . A computer implemented method of automatic classification of land use and land cover (LULC), comprising:
an unmanned aerial vehicle (UAV) including
capturing, by a sensor, a plurality of remote sensing (RS) images; and
transmitting, by UAV communication circuitry, the plurality of RS images;
a server computer including
receiving, by server communication circuitry, the plurality of RS images;
storing, by a memory, the plurality of RS images;
iteratively training, by a machine learning engine, a deep learning network including:
inputting, by an input layer, the plurality of RS images from the memory;
preventing, by at least one residual block, a training gradient from vanishing;
extracting, by a plurality of inception blocks, features of the RS images at multiple scales,
balancing, by a plurality of bottleneck blocks and inverted bottleneck blocks, complexity of computation;
promoting, by a plurality of dense blocks, reuse of the extracted features;
and
determining, by a softmax layer, at least one class label and a respective confidence score for the class label based on the extracted features.
12 . The computer implemented method of claim 11 , further comprising emphasizing, by an integrated at least one squeeze-excitation (SE) block, a region of a RS image.
13 . The computer implemented method of claim 12 , further comprising
passing, in each said SE block, input features through a global average pooling layer to perform a squeeze operation through which a scalar value is calculated for each channel;
reshaping a resulting squeezed vector to (1, 1,C), where C is a number of channels; and
applying two dense layers to the reshaped vector, wherein a first of the dense layers reduces the number of channels and a second of the dense layers restores the original number of channels.
14 . The computer implemented method of claim 13 , further comprising extracting, by each inception block embedded after a respective said SE block and having a plurality of parallel layers, different features for each said parallel layer having different kernel sizes of 1×1, 3×3, and 5×5.
15 . The computer implemented method of claim 11 , further comprising reducing, by a compression layer in each bottleneck block, spatial dimensions of the extracted features.
16 . The computer implemented method of claim 13 , further comprising:
reducing, by a first bottleneck block having a stride of 2, spatial dimensions; and
enhancing, by two other bottleneck blocks each having a stride of 1, feature extraction.
17 . The computer implemented method of claim 13 , wherein each dense block is connected to a respective said inverted bottleneck block, and
wherein each layer in a dense block is connected directly to all subsequent layers, and
the method further comprising, for each dense block:
performing a 1×1 convolution;
performing a 3×3 convolution;
concatenating results of the 1×1 convolution and the 3×3 convolution; and
emphasizing a region, by a respective squeeze and excitation block.
18 . The computer implemented method of claim 11 , further comprising applying, by the machine learning engine, a crossentropy loss function in order to train the deep learning network.
19 . The computer implemented method of claim 11 , wherein the unmanned aerial vehicle (UAV) further includes an on-board machine learning engine configured with the trained deep learning network, the method further comprising
classifying, in real-time, LULC, by
capturing, by the sensor every predetermined time period, a remote sensing (RS) image; and
classifying, by the on-board machine learning engine, the RS image at the predetermined time period.
20 . The computer implemented method of claim 19 , further comprising
transmitting, by the UAV communication circuitry, a plurality of RS images captured over a plurality of the predetermined time periods, together with a respective class and confidence score.