Input image size switchable network for adaptive runtime efficient image classification
Techniques related to implementing and training image classification networks are discussed. Such techniques include applying shared convolutional layers to input images regardless of resolution and applying normalization selectively based on the input image resolution. Such techniques further include training using mixed image size parallel training and mixed image size ensemble distillation.
1. A system for image classification, the system comprising:
a non-transitory memory to store a first image at a first resolution and a second image at a second resolution less than the first resolution; and
one or more processors coupled to the memory, the one or more processors to:
apply a convolutional neural network layer to the first image or first feature maps corresponding to the first image using a plurality of convolutional layer parameters to generate one or more second feature maps corresponding to the first image;
perform a first normalization on the one or more second feature maps using a plurality of first normalization parameters to generate one or more third feature maps;
generate a first label for the first image using the one or more third feature maps;
apply the convolutional neural network layer to the second image or fourth feature maps corresponding to the second image using the plurality of convolutional layer parameters to generate one or more fifth feature maps corresponding to the second image;
perform a second normalization on the one or more fifth feature maps using a plurality of second normalization parameters exclusive of the first normalization parameters to generate one or more sixth feature maps; and
generate a second label for the second image using the one or more sixth feature maps.
2. The system of claim 1 , wherein the one or more processors are to generate the first label by applying one or more additional convolutional neural network layers each followed by an additional first normalization and applying a fully connected layer, and wherein the one or more processors are to generate the second label by applying the one or more additional convolutional neural network layers each followed by additional second normalization using parameters exclusive of those used in the additional first normalizations and applying the fully connected layer.
3. The system of claim 2 , wherein the first normalization and additional first normalizations are selected in response to the first image being at the first resolution and the second normalization and additional second normalizations are selected in response to the second image being at the second resolution.
4. The system of claim 1 , wherein the convolutional layer parameters, the first normalization parameters, and the second normalization parameters include pre-trained convolutional neural network parameters trained based on:
generation of a training set including first and second training images at the first and second resolutions, respectively, and a corresponding ground truth label, the first and second training images corresponding to a same image instance; and
parameter adjustment, at a training iteration, of the pre-trained convolutional neural network parameters in parallel using a loss term including a sum of cross entropy losses based on application of the pre-trained convolutional neural network parameters to the first and second training images.
5. The system of claim 4 , wherein the parameter adjustment includes adjustment to fully connected layer parameters of the pre-trained convolutional neural network parameters, the convolutional layer parameters and fully connected layer parameters to be shared across input image sizes and the first and second normalization parameters to be unshared across the input image sizes.
6. The system of claim 1 , wherein the convolutional layer parameters, the first normalization parameters, and the second normalization parameters include pre-trained convolutional neural network parameters trained based on:
generation of a training set including first, second, and third training images at the first resolution, the second resolution, and a third resolution less than the second resolution, and a corresponding ground truth label, the first, second, and third training images corresponding to a same image instance;
generation, at a training iteration, of an ensemble prediction based on first, second, and third predictions made using the first, second, and third training images, respectively; and
comparison of the ensemble prediction to the ground truth label.
7. The system of claim 6 , wherein the ensemble prediction includes a weighted average of logits corresponding to the pre-trained convolutional neural network parameters as applied to each of the first, second, and third training images, the weighted average of the logits weighted using logit importance scores.
8. The system of claim 6 , wherein, at the training iteration, a parameters update is based on minimization of a loss function including a sum of an ensemble loss term based on a classification probability using the ensemble prediction and a distillation loss term based on a divergence of each the first, second, and third predictions from the ensemble prediction.
9. The system of claim 8 , wherein the distillation loss term includes a first divergence of the second prediction from the first prediction, a second divergence of the second prediction from the third prediction, and a third divergence of the third prediction from the first prediction.
10. The system of claim 8 , wherein the loss function includes a sum of cross entropy losses based on application of the pre-trained convolutional neural network parameters to the first, second, and third training images.
11. The system of claim 1 , wherein the convolutional neural network layer includes one of a pruned convolutional neural network layer or a quantized convolutional neural network layer.
12. A method for image classification, the method comprising:
receiving a first image at a first resolution and a second image at a second resolution less than the first resolution;
applying a convolutional neural network layer to the first image or first feature maps corresponding to the first image using a plurality of convolutional layer parameters to generate one or more second feature maps corresponding to the first image;
performing a first normalization on the one or more second feature maps using a plurality of first normalization parameters to generate one or more third feature maps;
generating a first label for the first image using the one or more third feature maps;
applying the convolutional neural network layer to the second image or fourth feature maps corresponding to the second image using the plurality of convolutional layer parameters to generate one or more fifth feature maps corresponding to the second image;
performing a second normalization on the one or more fifth feature maps using a plurality of second normalization parameters exclusive of the first normalization parameters to generate one or more sixth feature maps; and
generating a second label for the second image using the one or more sixth feature maps.
13. The method of claim 12 , wherein generating the first label includes applying one or more additional convolutional neural network layers each followed by an additional first normalization and applying a fully connected layer, and wherein the generating of the second label includes applying the one or more additional convolutional neural network layers each followed by additional second normalization using parameters exclusive of those used in the additional first normalizations and applying the fully connected layer.
14. The method of claim 12 , wherein the convolutional layer parameters, the first normalization parameters, and the second normalization parameters include pre-trained convolutional neural network parameters trained based on:
generation of a training set including first and second training images at the first and second resolutions, respectively, and a corresponding ground truth label, the first and second training images corresponding to a same image instance; and
parameter adjustment, at a training iteration, of the pre-trained convolutional neural network parameters in parallel using a loss term including a sum of cross entropy losses based on application of the pre-trained convolutional neural network parameters to the first and second training images.
15. The method of claim 12 , wherein the convolutional layer parameters, the first normalization parameters, and the second normalization parameters include pre-trained convolutional neural network parameters trained based on:
generation of a training set including first, second, and third training images at the first resolution, the second resolution, and a third resolution less than the second resolution, and a corresponding ground truth label, the first, second, and third training images corresponding to a same image instance;
generation, at a training iteration, of an ensemble prediction based on first, second, and third predictions made using the first, second, and third training images, respectively; and
comparison of the ensemble prediction to the ground truth label.
16. The method of claim 15 , wherein, at the training iteration, a parameters update is based on minimization of a loss function including a sum of an ensemble loss term based on a classification probability using the ensemble prediction and a distillation loss term based on a divergence of each the first, second, and third predictions from the ensemble prediction.
17. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a device, cause the device to at least:
receive a plurality of sequential video images including representations of a human face;
receive a first image at a first resolution and a second image at a second resolution less than the first resolution;
apply a convolutional neural network layer to the first image or first feature maps corresponding to the first image using a plurality of convolutional layer parameters to generate one or more second feature maps corresponding to the first image;
perform a first normalization on the one or more second feature maps using a plurality of first normalization parameters to generate one or more third feature maps;
generate a first label for the first image using the one or more third feature maps;
apply the convolutional neural network layer to the second image or fourth feature maps corresponding to the second image using the plurality of convolutional layer parameters to generate one or more fifth feature maps corresponding to the second image;
perform a second normalization on the one or more fifth feature maps using a plurality of second normalization parameters exclusive of the first normalization parameters to generate one or more sixth feature maps; and
generate a second label for the second image using the one or more sixth feature maps.
18. The machine readable medium of claim 17 , wherein the instructions are to cause the device to generate the first label by applying one or more additional convolutional neural network layers each followed by an additional first normalization and applying a fully connected layer, and wherein the instructions are to cause the device to generate the second label by applying the one or more additional convolutional neural network layers each followed by additional second normalization using parameters exclusive of those used in the additional first normalizations and applying the fully connected layer.
19. The machine readable medium of claim 17 , wherein the convolutional layer parameters, the first normalization parameters, and the second normalization parameters include pre-trained convolutional neural network parameters trained based on:
generation of a training set including first and second training images at the first and second resolutions, respectively, and a corresponding ground truth label, the first and second training images corresponding to a same image instance; and
parameter adjustment, at a training iteration, of the pre-trained convolutional neural network parameters in parallel using a loss term including a sum of cross entropy losses based on application of the pre-trained convolutional neural network parameters to the first and second training images.
20. The machine readable medium of claim 17 , wherein the convolutional layer parameters, the first normalization parameters, and the second normalization parameters include pre-trained convolutional neural network parameters trained based on:
generation of a training set including first, second, and third training images at the first resolution, the second resolution, and a third resolution less than the second resolution, and a corresponding ground truth label, the first, second, and third training images corresponding to a same image instance;
generation, at a training iteration, of an ensemble prediction based on first, second, and third predictions made using the first, second, and third training images, respectively; and
comparison of the ensemble prediction to the ground truth label.
21. The machine readable medium of claim 20 , wherein, at the training iteration, a parameters update is based on minimization of a loss function including a sum of an ensemble loss term based on a classification probability using the ensemble prediction and a distillation loss term based on a divergence of each the first, second, and third predictions from the ensemble prediction.