Multi-layer fusion in a convolutional neural network for image classification
A method and system for domain adaptation based on multi-layer fusion in a convolutional neural network architecture for feature extraction and a two-step training and fine-tuning scheme. The architecture concatenates features extracted at different depths of the network to form a fully connected layer before the classification step. First, the network is trained with a large set of images from a source domain as a feature extractor. Second, for each new domain (including the source domain), the classification step is fine-tuned with images collected from the corresponding site. The features from different depths are concatenated with and fine-tuned with weights adjusted for a specific task. The architecture is used for classifying high occupancy vehicle images.
1. A method of training a convolutional neural network (CNN) for domain
adaptation utilizing features extracted from multiple levels, including:
selecting a CNN architecture including a plurality of convolutional layers and fully connected layers;
training the CNN on a source domain data set;
selecting a plurality of layers from the plurality of convolutional layers across the trained CNN;
extracting features from the selected layers from the trained CNN;
concatenating the extracted features to form a feature vector;
connecting the feature vector to a fully connected neural network classifier; and,
fine-tuning the fully connected neural network classifier from a target domain data set
by optimizing weights of the CNN with respect to the target domain data set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN.
2. The method of claim 1 wherein the selected CNN network can be any CNN network selected from the group consisting of GoogLeNet, AlexNet, VGG-M, VGG-D, and VGG-F.
3. The method of claim 1 wherein the concatenating features includes a feature vector that can be weighted before the concatenating using a learning rate.
4. The method of claim 1 wherein the connecting the concatenated feature vector includes any type of fully connected neural network including softmax activation.
5. The method of claim 1 wherein the target domain can be different or the same as the source domain.
6. The method of claim 1 wherein the training and fine-tuning use different datasets.
7. An image classification system comprising:
a computer programmed to perform classification of an input image from a target domain by operations including:
processing the input image using a convolutional neural network (CNN) having a plurality of network layers and trained on a source domain training set;
processing outputs of at least a fraction of the plurality of network layers of the CNN using a features fusion network trained on a target domain training set to generate a classification of the input image;
training the CNN by optimizing weights of the CNN with respect to the source domain training set and training the combination of the CNN and the features fusion network by optimizing weights of the features fusion network with respect to the target domain training set wherein the features fusion network includes:
a features extraction layer operating to extract features from the network layers of the CNN;
a concatenation layer that concatenates the extracted features to generate a concatenated features vector representation of the input image;
and wherein the weights of the features fusion network include weights of the extracted features in the concatenated features vector; and,
optimizing weights of the CNN with respect to the target domain training set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN.
8. A method of adapting a convolutional neural network (CNN) trained to classify images of a source domain to a target domain, the adaptation method comprising:
inputting features output by at least a fraction of the network levels of the CNN into a features fusion network outputting a weighted combination of the inputted features; and
training weights of the weighted combination of inputted features using images in a target domain different from the source domain; and
classifying the image in accordance with the trained weights by optimizing weights of the CNN with respect to the target domain data set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN.
9. The adaptation method of claim 8 further comprising:
training at least some weights of the CNN using the images in a target domain different from the source domain concurrently with the training of the weights of the weighted combination of inputted features.
10. The adaptation method of claim 9 wherein the training of at least some weights of the CNN preferentially adjusts weights of higher network levels of the CNN.
11. The method of claim 1 further including confirming vehicle occupancy in a high occupancy vehicle lane by using the trained CNN.
12. The system of claim 7 wherein the operations further include confirming vehicle occupancy in a high occupancy vehicle lane by using the trained CNN.
13. The method of claim 8 further including confirming vehicle occupancy in a high occupancy vehicle lane by using the trained CNN.
14. The system of claim 7 wherein the features fusion network includes:
a features extraction layer operating to extract features from the network layers of the CNN; and
a concatenation layer that concatenates the extracted features to generate a concatenated features vector representation of the input image.
15. The system of claim 7 wherein the concatenation layer further includes a nonlinear activation function.
16. The system of claim 7 wherein the features extraction layer comprises a
sequence of layers including, in order:
an average pooling layer;
a convolution layer; and
one or more fully connected layers.