Convolutional neural network (CNN) for automatic target recognition in a satellite
Methods and structures are presented for implementing an automatic target recognition system as a convolutional neural network (CNN) in a satellite or other environment with constrained resources, such as limited memory capacity and limited processing capability. For example, this allows for the automatic target recognition to be implemented on a field programmable gate array (FPGA). Image data is split into subsets of contiguous pixels, with the subsets processed in parallel in a CNN of a corresponding processing node using quantized weight values that are determined in a training process that accounts for the constraints of the automatic target recognition system. The results of the automatic target recognition process is based on the combined output of the processing nodes.
1. A satellite, comprising:
an image sensor configured to generate image data of a two dimensional array of pixel values; and
an automatic target recognition circuit configured to receive the image data and comprising:
a plurality of N processing nodes each configured to apply a convolutional neural network (CNN) to a subset of pixels values of the image data received from the image sensor; and
one or more control circuits configured to:
receive the image data from the image sensor;
separate the image data into a plurality of N non-overlapping subsets of the image data, each of the subsets comprising pixel values of a plurality of contiguous pixel locations, the combined pixel locations of the N subsets containing all pixel locations of the image data;
process in parallel each of the N subsets of the image data in a corresponding one of the processing nodes by applying the processing node's CNN to the corresponding subset of the image data;
combine results of the processing of each of the N subsets of the image data in a corresponding one of the processing nodes to obtain a combined output for the image data; and
determine whether a target is recognized based upon the combined output for the image data.
2. The satellite of claim 1 , wherein the automatic target recognition circuit is a field programmable gate array.
3. The satellite of claim 2 , further comprising:
an antenna configured to receive instructions to configure the field programmable gate array while the satellite is on-orbit,
and wherein the one or more control circuits configured to:
configure the field programmable gate array according to the instructions.
4. The satellite of claim 1 , wherein receiving the image data from the image sensor comprises streaming the subsets of image data.
5. The satellite of claim 1 , wherein the automatic target recognition circuit further comprises:
a buffer memory configured to store the separated the image data.
6. The satellite of claim 1 , wherein the one or more control circuits include:
a broadcaster configured to transmit to each of the subsets of the image data to a corresponding processing node.
7. The satellite of claim 6 , wherein the one or more control circuits include:
a micro sequencer configured to coordinate with the broadcaster the transmission of the subsets of the image data and instructions to the processing nodes.
8. The satellite of claim 7 , wherein the broadcaster and the micro sequencer are configured to interleave the broadcast of the subsets of the image data and the instructions to the processing nodes.
9. The satellite of claim 1 , wherein the automatic target recognition circuit further comprises:
a memory configured to store weight values for the CNNs.
10. The satellite of claim 9 , wherein the weight values are quantized values.
11. The satellite of claim 9 , further comprising:
an antenna configured to receive the weight values while the satellite is on-orbit,
and wherein the one or more control circuits configured to:
store the received weight values in the memory.
12. The satellite of claim 11 , wherein storing the received weight values in the memory includes updating previously stored weight values.
13. The satellite of claim 1 , wherein each of the processing nodes has a corresponding random access memory configured to store intermediate computational values when applying the CNN to the corresponding subset of pixels values of the image data.
14. The satellite of claim 1 , wherein each of the N processing nodes is configured to determine a corresponding intermediate result, the automatic target recognition circuit further comprising:
one or more shared CNN layers configured to receive the N intermediate results as input and determine a target recognition result from a combination of the N intermediate results.
15. The satellite of claim 14 , wherein the one or more shared layers include a softmax layer.
16. The satellite of claim 14 , wherein the one or more shared layers include a sigmoid layer.
17. The satellite of claim 1 , wherein each of the subsets of pixel values correspond to an M by M square of contiguous pixel locations, where M is an integer greater than 1.
18. The satellite of claim 17 , wherein the digital sensor is a panchromatic sensor.
19. A method, comprising:
receiving instructions for configuring a field programmable gate array (FPGA) as an automatic target recognition circuit;
configuring the FPGA according to the instructions to include:
a memory; and
a plurality of N processing nodes each configured to apply a convolutional neural network (CNN) to image data;
receiving weight values for the CNNs;
storing the weight values in the memory;
subsequent to configuring the FPGA and storing the weight values, receiving image data from a two dimensional array of pixel values;
separating the image data into a plurality of N non-overlapping subsets of the image data, each of the subsets comprising pixel values of a plurality of contiguous pixel locations, the combined pixel locations of the N subsets containing all pixel locations of the image data;
processing in parallel each of the N subsets of the image data in a corresponding one of the processing nodes by applying the processing node's CNN using the stored weight values to the corresponding subset of image data;
combining results of the processing of each of the N subsets of the image data in a corresponding one of the processing nodes to obtain a combined output for the image data; and
determining whether a target is recognized based upon the combined output for the image data.
20. An apparatus, comprising:
a memory;
a plurality of N processing nodes each configured to apply a corresponding convolutional neural network (CNN) to image data;
one or more shared neural network layers; and
one or more control circuits configured to:
receive weight values for the CNNs;
store the weight values in the memory;
receive image data from a two dimensional array of pixel values;
separate the image data into a plurality of N non-overlapping subsets of the image data, each of the subsets comprising pixel values of a plurality of contiguous pixel locations, the combined pixel locations of the N subsets containing all pixel locations of the image data;
process in parallel each of the N subsets of the image data in a corresponding one of the processing nodes by applying the processing node's CNN using the stored weight values to the corresponding subset of image data to generate N intermediate results;
receive the N intermediate results as input to the one or more shared neural network layers;
combine results of the processing of each of the N subsets of the image data in a corresponding one of the processing nodes to obtain a combined output for the image data; and
determine whether a target is recognized based upon the combined output for the image data.
21. The method of claim 19 , further comprising prior to configuring the FPGA:
selecting the value of N and the number of pixels in the non-overlapping subsets;
determining an accuracy of the CNN using the selected value of N and selected number of pixels in the non-overlapping subsets; and
based on the determined accuracy, adjusting the selected value of N and selected number of pixels in the non-overlapping subsets.