Systems and methods for neural network pruning with accuracy preservation
Training a neural network (NN) may include training a NN N, and for S, a version of N to be sparsified (e.g. a copy of N), removing NN elements from S to create a sparsified version of S, and training S using outputs from N (e.g. “distillation”). A boosting or reintroduction phase may follow sparsification: training a NN may include for a trained NN N and S, a sparsified version of N, re-introducing NN elements previously removed from S, and training S using outputs from N. The boosting phase need not use a NN sparsified by “distillation.” Training and sparsification, or training and reintroduction, may be performed iteratively or over repetitions.
1. A method for training a neural network (NN), the method comprising:
training a NN N; and
for S, a version of N to be sparsified:
removing NN elements from S to create a sparsified version of S; and
training S using outputs from N.
2. The method of claim 1 wherein the outputs from N comprise the outputs of a softmax layer.
3. The method of claim 1 wherein the NN elements comprise links.
4. The method of claim 1 comprising performing the removing and training operations over a series of iterations.
5. The method of claim 1 comprising training S using, for an input, outputs from N based on that input and the ground truth for that input.
6. The method of claim 1 comprising re-introducing NN elements previously removed from S, and training S using outputs from N.
7. The method of claim 1 , comprising training S using a loss generated from outputs from N.
8. A method for training a neural network (NN), the method comprising:
for a trained NN N and S, a sparsified version of N:
re-introducing NN elements previously removed from S; and
training S using outputs from N.
9. The method of claim 8 wherein the outputs from N comprise the outputs of a softmax layer.
10. The method of claim 8 wherein the NN elements comprise links.
11. The method of claim 8 comprising performing the re-introducing and training operations over a series of iterations.
12. The method of claim 8 comprising training S using, for an input, outputs from N based on that input and the ground truth for that input.
13. The method of claim 8 wherein S is sparsified by training S using outputs from N.
14. The method of claim 8 , comprising training S using a loss generated from outputs from N.
15. A system for training a neural network (NN), the system comprising:
a memory, and
a processor configured to:
train a NN N; and
for S, a version of N to be sparsified:
remove NN elements from S to create a sparsified version of S; and
train S using outputs from N.
16. The system of claim 15 wherein the outputs from N comprise the outputs of a softmax layer.
17. The system of claim 15 wherein the NN elements comprise links.
18. The system of claim 15 wherein the processor is configured to perform the removing and training operations over a series of iterations.
19. The system of claim 15 wherein the processor is configured to train S using, for an input, outputs from N based on that input and the ground truth for that input.
20. The system of claim 15 wherein the processor is configured to re-introduce NN elements previously removed from S, and training S using outputs from N.
21. The system of claim 15 , wherein the processor is configured to train S using a loss generated from outputs from N.
22. A system for training a neural network (NN), the system comprising:
a memory, and
a processor configured to:
for a trained NN N and S, a sparsified version of N:
re-introduce NN elements previously removed from S; and
train S using outputs from N.
23. The system of claim 22 wherein the outputs from N comprise the outputs of a softmax layer.
24. The system of claim 22 wherein the NN elements comprise links.
25. The system of claim 22 wherein the processor is configured to perform the re-introducing and training operations over a series of iterations.
26. The system of claim 22 wherein the processor is configured to train S using, for an input, outputs from N based on that input and the ground truth for that input.
27. The system of claim 22 wherein S is sparsified by training S using outputs from N.
28. The system of claim 22 , wherein the processor is configured to train S using a loss generated from outputs from N.