System and method for domain specific neural network pruning
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for domain-specific pruning of neural networks are described. An exemplary method includes obtaining a first neural network trained based on a first training dataset; obtaining one or more second training datasets respectively from one or more domains; training, based on the first neural network and the one or more second training datasets, a second neural network comprising the first neural network and one or more branches extended from the first neural network. The one or more branches respectively correspond to the one or more domains, and each comprises one or more layers trained based on one of the one or more second training datasets. The method may further include: pruning the second neural network by reducing a number of active neurons; and applying the pruned second neural network for inferencing in the one or more domains.
1. A computer-implemented method, comprising:
obtaining a first neural network trained based on a first training dataset;
obtaining one or more second training datasets respectively from one or more domains;
training, based on the first neural network and the one or more second training datasets, a second neural network comprising the first neural network and one or more branches extended from the first neural network, wherein the one or more branches respectively correspond to the one or more domains, and each comprises one or more layers trained based on one of the one or more second training datasets;
pruning the second neural network by reducing a number of active neurons; and
applying the pruned second neural network for inferencing in the one or more domains.
2. The method of claim 1 , wherein the training the second neural network comprises:
training the second neural network based on the first neural network and the one or more second training datasets until an objective function converges; and
the pruning the second neural network comprises:
pruning the second neural network until reaching a sparsity ratio.
3. The method of claim 2 , wherein the training the second neural network further comprises:
increasing the sparsity ratio; and
further training the pruned second neural network based on the one or more second training datasets until the objective function converges; and
further pruning the further trained second neural network to reach the increased sparsity ratio until an exit condition is met.
4. The method of claim 3 , wherein the exit condition comprises at least one of the following: a number of iterations, and the sparsity ratio has reached a target sparsity ratio.
5. The method of claim 1 , wherein each of the one or more second training datasets comprises a plurality of supervised data records collected from a corresponding domain.
6. The method of claim 1 , further comprising:
applying the first neural network for object detections in a domain that is different from the one or more domains.
7. The method of claim 1 , wherein the applying the second neural network for object detections in the one or more domains comprises:
receiving an object detection request comprising a domain identifier;
determining one of the second neural network based on the domain identifier; and
performing object detection based on the one second neural network.
8. The method of claim 1 , wherein the one or more second training datasets comprise images collected by one or more cameras representing the one or more domains, and each of the images comprises a camera identifier and at least one label.
9. The method of claim 1 , wherein the training the second neural network comprising the first neural network and one or more branches comprises:
for each of the one or more second training datasets, extending the first neural network by creating a branch with one or more layers trained based on the second training dataset.
10. The method of claim 1 , further comprising:
pruning the first neural network before the training of the second neural network;
wherein the training of the second neural network comprises:
training the second neural network based on the pruned first neural network and the one or more second training datasets.
11. A system comprising one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations comprising:
obtaining a first neural network trained based on a first training dataset;
obtaining one or more second training datasets respectively from one or more domains;
training, based on the first neural network and the one or more second training datasets, a second neural network comprising the first neural network and one or more branches extended from the first neural network, wherein the one or more branches respectively correspond to the one or more domains, and each comprises one or more layers trained based on one of the one or more second training datasets;
pruning the second neural network by reducing a number of active neurons; and
applying the pruned second neural network for inferencing in the one or more domains.
12. The system of claim 11 , wherein the training the second neural network comprises:
training the second neural network based on the first neural network and the one or more second training datasets until an objective function converges; and
the pruning the second neural network comprises:
pruning the second neural network until reaching a sparsity ratio.
13. The system of claim 12 , wherein the training the second neural network further comprises:
increasing the sparsity ratio; and
further training the pruned second neural network based on the one or more second training datasets until the objective function converges; and
further pruning the further trained second neural network to reach the increased sparsity ratio until an exit condition is met.
14. The system of claim 11 , wherein the applying the second neural network for object detections in the one or more domains comprises:
receiving an object detection request comprising a domain identifier;
determining one of the second neural network based on the domain identifier; and performing object detection based on the one second neural network.
15. The system of claim 11 , wherein the operations further comprise:
pruning the first neural network before the training of the second neural network;
wherein the training of the second neural network comprises:
training the second neural network based on the pruned first neural network and the one or more second training datasets.
16. A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
obtaining a first neural network trained based on a first training dataset;
obtaining one or more second training datasets respectively from one or more domains;
training, based on the first neural network and the one or more second training datasets, a second neural network comprising the first neural network and one or more branches extended from the first neural network, wherein the one or more branches respectively correspond to the one or more domains, and each comprises one or more layers trained based on one of the one or more second training datasets;
pruning the second neural network by reducing a number of active neurons; and
applying the pruned second neural network for inferencing in the one or more domains.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the training the second neural network comprises:
training the second neural network based on the first neural network and the one or more second training datasets until an objective function converges; and
the pruning the second neural network comprises:
pruning the second neural network until reaching a sparsity ratio.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the training the second neural network further comprises:
increasing the sparsity ratio; and
further training the pruned second neural network based on the one or more second training datasets until the objective function converges; and
further pruning the further trained second neural network to reach the increased sparsity ratio until an exit condition is met.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the applying the second neural network for object detections in the one or more domains comprises:
receiving an object detection request comprising a domain identifier;
determining one of the second neural network based on the domain identifier; and performing object detection based on the one second neural network.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the operations further comprise:
pruning the first neural network before the training of the second neural network;
wherein the training of the second neural network comprises:
training the second neural network based on the pruned first neural network and the one or more second training datasets.