Joint training of neural networks using multi scale hard example mining
An example apparatus for mining multi-scale hard examples includes a convolutional neural network to receive a mini-batch of sample candidates and generate basic feature maps. The apparatus also includes a feature extractor and combiner to generate concatenated feature maps based on the basic feature maps and extract the concatenated feature maps for each of a plurality of received candidate boxes. The apparatus further includes a sample scorer and miner to score the candidate samples with multi-task loss scores and select candidate samples with multi-task loss scores exceeding a threshold score.
1. An apparatus for mining multi-scale hard examples, comprising:
a convolutional neural network to receive a mini-batch of sample candidates and generate basic feature maps;
a feature extractor and combiner to generate concatenated feature maps based on the basic feature maps and extract the concatenated feature maps for each of a plurality of received candidate boxes; and
a sample scorer and miner to score the candidate samples with multi-task loss scores and select candidate samples with multi-task loss scores exceeding a threshold score.
2. The apparatus of claim 1 , wherein the candidate boxes are to be generated by a region proposal network in response to receiving a plurality of mini-batches of example images.
3. The apparatus of claim 1 , wherein each of the concatenated feature maps comprise a plurality of channels comprising the basic feature maps resized to a reference layer size.
4. The apparatus of claim 1 , wherein the multi-task loss score for each candidate sample is to be calculated based on a localization score and a classification score corresponding to classification and localization losses calculated for each candidate sample in a respective Stochastic Gradient Descent (SGD).
5. The apparatus of claim 1 , wherein the convolutional neural network is to resize sample candidates from the received mini-batch into a standard scale.
6. The apparatus of claim 1 , wherein the feature extractor and combiner is to select a reference layer in the convolutional neural network and up-sample or down-sample feature maps from other layers in the convolutional neural network to generate the concatenated feature maps comprising a feature map of the reference layer and the up-sampled or down-sampled feature maps of other layers in the convolutional neural network.
7. The apparatus of claim 1 , wherein the mini-batch comprises negative example images and positive example images to be used for training.
8. The apparatus of claim 1 , wherein the sample scorer and miner comprises a network of fully-connected layers.
9. The apparatus of claim 1 , wherein a predetermined number of the selected sample candidates are iteratively grouped for back-propagating and updating a detection network.
10. The apparatus of claim 1 , wherein the selected sample candidates are to be used to jointly train a region proposal network and a detection network.
11. A method for training a neural network, comprising:
pretraining, via a processor, a CNN model to initialize base networks, the base networks comprising a region proposal network and a detection network;
training, via the processor, the region proposal network initialized with the pretrained CNN model to generate region proposals;
training, via the processor, the detection network initialized with the pretrained CNN model with the region proposals using multi-scale hard example mining to train feature layers to detect features;
fine-tuning, via the processor, the region proposal network using the trained feature layers to generate tuned region proposals and modified feature layers; and
training, via the processor, the detection network using multi-scale hard example mining based on the tuned region proposals and the modified feature layers.
12. The method of claim 11 , wherein multi-scale hard example mining comprises selecting hard examples within a received mini-batch of example images based on classification and localization losses calculated in a respective Stochastic Gradient Descent (SGD) for feature maps of each scale.
13. The method of claim 11 , wherein multi-scale hard example mining comprises generating concatenated feature maps and extracting the concatenated feature maps for a plurality of received candidate boxes from the region proposal network.
14. The method of claim 11 , wherein pretraining the CNN model comprises using a large scale object classification dataset.
15. The method of claim 11 , wherein fine-tuning the region proposal network comprises back-propagating parameter values based on generated feature layers from an initialized detection network.
16. The method of claim 11 , wherein training region proposal network comprises training the region proposal network to calculate objectness scores for a plurality of candidate boxes and select candidate boxes with objectness scores above a threshold score to generate the region proposals.
17. The method of claim 11 , wherein the tuned region proposals comprise mined hard examples.
18. The method of claim 11 , wherein the trained feature layers comprise shared layers.
19. The method of claim 11 , wherein training the region proposal network comprises training the region proposal network to generate a plurality of candidate boxes with a plurality of sizes and a plurality of scales based on a reference layers in the pretrained CNN model.
20. The method of claim 11 , wherein pretraining the CNN model comprises using mini-batches of example images comprising positive examples and negative examples.
21. At least one non-transitory computer readable medium for training a neural network having instructions stored therein that, in response to being executed on a computing device, cause the computing device to: pretrain a convolutional neural network (CNN) model to initialize base networks comprising a region proposal network and a detection network; train a region proposal network initialized with the pretrained CNN model to generate region proposals; train a detection network initialized with the pretrained CNN model with the region proposals using multi-scale hard example mining to train feature layers to detect features; fine-tune the region proposal network using the trained feature layers to generate tuned region proposals and modified feature layers; and train the detection network using multi-scale hard example mining based on the tuned region proposals and the modified feature layers.
22. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to receive a plurality of candidate boxes and sub-sample the candidate boxes based on loss values calculated for each of the candidate boxes.
23. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to receive a plurality of candidate boxes and select a fixed number of high-loss examples from the plurality of candidate boxes to be used to execute a back-propagation and fine-tune parameters of the detection network.
24. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to select a reference layer in the CNN model and up-sample or down-sample feature maps from other layers in the CNN model to generate a concatenated feature map comprising a feature map of the reference layer and the up-sampled or down-sampled feature maps of other layers in the CNN model.
25. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to resize example images from a received mini-batch into a standard scale.