Embedded semantic division network apparatus optimized for MMA that classifies pixels in vehicle images
Provided is an embedded semantic division network including a communication module configured to receive an image captured by a camera, a memory configured to store a semantic division network (MMANet)-based program for extracting a context of the captured image, and a processor extracts the context of the captured image by selecting a convolutional neural network (CNN) processing module or a depth-wise separable convolution (DSC) processing module according to a size of a activation map in each layer of the semantic division network that includes an encoder unit and a decoder unit including at least one of the CNN processing module and the DSC processing module that are connected from an upper layer to a lower layer and reduce features of an input image.
1. An embedded semantic division network apparatus embedded in a semantic division network and optimized for a matrix multiplication accelerator (MMA) that classifies pixels in vehicle images, the semantic division network including an encoder unit, a decoder unit, and a plurality of layers including upper and lower layers, the encoder and decoder units being provided throughout the plurality of layers, each of the encoder unit and decoder unit comprising at least one of a convolutional neural network (CNN) processing module and a depth-wise separable convolution (DSC) processing module that are configured to reduce some features of an image, the embedded semantic division network apparatus comprising:
a processor; and
a computer-readable medium in communication with the processor and storing instructions that, when executed by the processor, cause the processor to control the embedded semantic division network apparatus to perform:
receiving an input image captured by a camera;
selecting one of the CNN processing module and the DSC processing module based on a size of an activation map in each layer of the semantic division network; and
extracting, using the selected one of the CNN processing module and the DSC processing module, a context of the input image.
2. The embedded semantic division network apparatus of claim 1 , wherein, for extracting the context of the input image, the instructions, when executed by the processor, further cause the processor to control the embedded semantic division network apparatus to perform:
receiving feature information of the image output from the encoder unit through extended Atrous spatial pyramid pooling (ASPP) applied to a predetermined layer of the decoder unit; and
extracting the feature information corresponding to the encoded image.
3. The embedded semantic division network apparatus of claim 2 , wherein the extended ASPP includes a plurality of ASPPs configured to extract a high-quality context using a reconstructed shape without global average pooling paths, the plurality of ASPPs including a first ASPP applied to the upper layer and a second ASPP applied to the lower layer.
4. The embedded semantic division network apparatus of claim 3 , wherein the second ASPP is applied to a lowest one of the plurality of layers of the embedded semantic division network.
5. The embedded semantic division network apparatus of claim 3 , wherein the second ASPP includes:
an input stage including (1) a plurality of CNNs configured to receive the feature information from the encoder unit and (2) an extended path arranged in parallel with the plurality of CNNs, and
an output stage configured to combine the feature information received by the input stage and input the combined feature information to the CNN.
6. The embedded semantic division network apparatus of claim 5 , wherein the extended path includes:
the CNN;
a plurality of DSCs configured to receive an output of the CNN; and
a bilinear interpolation unit configured to combine a plurality of outputs from the DSCs and bilinearly interpolate the combined output.
7. The embedded semantic division network apparatus of claim 3 , wherein the first ASPP includes:
an input stage including the CNN configured to receive the feature information from the second ASPP and a plurality of inverse DSCs (IDSCs) arranged in parallel with the CNN; and
an output stage configured to combine the feature information received by the input stage and to input the combined feature information to the CNN.
8. The embedded semantic division network apparatus of claim 1 , wherein the encoder unit includes a shape information transfer unit including one or more CNNs provided in a predetermined one of the plurality of layers and configured to transmit, to the decoder unit, detailed shape information of the input image corresponding to each of the layers.
9. The embedded semantic division network apparatus of claim 1 , wherein:
the plurality of layers includes first to fourth layers,
the encoder unit includes (1) two first modules provided in the third layer configured to abstract feature information from a previous layer and (2) a second module provided in the fourth layer,
each first module includes (1) a plurality of DSCs configured to receive the feature information and having different dilations, (2) a pointwise convolution unit configured to combine the feature information received by the plurality of DSCs, and (3) a first summer configured to sum the feature information received by the DSCs and the combined feature information from the pointwise convolution unit, and
the second module includes (1) a plurality of CNN layers configured to receive the feature information, and (2) a second summer configured to sum a plurality of outputs from the plurality of CNN layers and the feature information.
10. The embedded semantic division network apparatus of claim 1 , wherein, in the embedded semantic division network, a maximum number of channels is limited to 64.