Determining layer ranks for compression of deep networks
An apparatus of operating a computational network is configured to determine a low-rank approximation for one or more layers of the computational network based at least in part on a set of residual targets. A set of candidate rank vectors corresponding to the set of residual targets may be determined. Each of the candidate rank vectors may be evaluated using an objective function. A candidate rank vector may be selected and used to determine the low rank approximation. The computational network may be compressed based on the low-rank approximation. In turn the computational network may be operated using the one or more compressed layers.
1. A method of operating a computational network, comprising:
determining a low-rank approximation for one or more layers of the computational network based at least in part on a set of targets respectively corresponding to a set of tensor approximation residuals, and a rank vector of a set of candidate rank vectors associated with the set of tensor approximation residuals;
compressing at least one layer of the computational network based at least in part on the low-rank approximation; and
operating the computational network using the at least one compressed layer.
2. The method of claim 1 wherein, the low-rank approximation is automatically determined based on a performance metric.
3. The method of claim 2 , wherein the performance metric includes at least one of an accuracy metric, a completion time metric, and a complexity metric.
4. The method of claim 1 , wherein the compressing the at least one layer includes updating a bias associated with the at least one layer.
5. The method of claim 4 , wherein, the bias is updated by applying a vector m to compensate for a mean shift in an output activation of the at least one compressed layer and an output activation of the at least one layer.
6. The method of claim 1 , wherein the low-rank approximation is determined without fine tuning.
7. The method of claim 1 , wherein the low-rank approximation is determined using singular value decomposition.
8. The method of claim 1 , wherein the determining comprises:
determining the set of candidate rank vectors, wherein the set of candidate rank vectors satisfy each target of the set of targets;
evaluating each candidate rank vector of the set of candidate rank vectors; and
selecting the rank vector of the set of candidate rank vectors based on evaluations of a minimization function using the set of candidate rank vectors.
9. An apparatus of operating a computational network, comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor being configured to:
determine a low-rank approximation for one or more layers of the computational network based at least in part on a set of targets respectively corresponding to a set of tensor approximation residuals, and a rank vector of a set of candidate rank vectors associated with the set of tensor approximation residuals;
compress at least one layer of the computational network based at least in part on the low-rank approximation; and
operate the computational network using the at least one compressed layer.
10. The apparatus of claim 9 wherein, the at least one processor is further configured to determine the low-rank approximation automatically based on a performance metric.
11. The apparatus of claim 10 , wherein the performance metric includes at least one of an accuracy metric, a completion time metric, and a complexity metric.
12. The apparatus of claim 9 , wherein the at least one processor is further configured to update a bias associated with the at least one layer.
13. The apparatus of claim 12 , wherein, the at least one processor is further configured to update the bias by applying a vector m to compensate for a mean shift in an output activation of the at least one compressed layer and an output activation of the at least one layer.
14. The apparatus of claim 9 , wherein the at least one processor is further configured to determine the low-rank approximation without fine tuning.
15. The apparatus of claim 9 , wherein the at least one processor is further configured to determine the low-rank approximation using singular value decomposition.
16. The apparatus of claim 9 , wherein the at least one processor is further configured to:
determine the set of candidate rank vectors, wherein the set of candidate rank vectors satisfy each target of the set of targets;
evaluate each candidate rank vector of the set of candidate rank vectors; and
select the rank vector of the set of candidate rank vectors based on evaluations of a minimization function using the set of candidate rank vectors.
17. An apparatus for operating a computational network, comprising:
means for determining a low-rank approximation for one or more layers of the computational network based at least in part on a set of targets respectively corresponding to a set of tensor approximation residuals, and a rank vector of a set of candidate rank vectors associated with the set of tensor approximation residuals;
means for compressing at least one layer of the computational network based at least in part on the low-rank approximation; and
means for operating the computational network using the at least one compressed layer.
18. The apparatus of claim 17 , wherein the low-rank approximation is automatically determined based on a performance metric.
19. The apparatus of claim 18 , wherein the performance metric includes at least one of an accuracy metric, a completion time metric, and a complexity metric.
20. The apparatus of claim 17 , further comprising means for updating a bias associated with the at least one layer.
21. The apparatus of claim 17 , further comprising means for applying a vector m to compensate for a mean shift in an output activation of the at least one compressed layer and an output activation of the at least one layer.
22. The apparatus of claim 17 , further comprising means for determining the low-rank approximation without fine tuning.
23. The apparatus of claim 17 , further comprising means for determining the low-rank approximation using singular value decomposition.
24. The apparatus of claim 17 , further comprising:
means for determining set of candidate rank vectors, wherein the set of candidate rank vectors satisfy each target of the set of targets;
means for evaluating each candidate rank vector of the set of candidate rank vectors; and
means for selecting the rank vector of the set of candidate rank vectors based on evaluations of a minimization function using the set of candidate rank vectors.
25. A non-transitory, computer readable medium having executable code for operating a computational network, comprising code to:
determine a low-rank approximation for one or more layers of the computational network based at least in part on a set of targets respectively corresponding to a set of tensor approximation residuals, and a rank vector of a set of candidate rank vectors associated with the set of tensor approximation residuals;
compress at least one layer of the computational network based at least in part on the low-rank approximation; and
operate the computational network using the at least one compressed layer.
26. The non-transitory, computer readable medium of claim 25 , further comprising code to determine the low-rank approximation automatically based on a performance metric, the performance metric comprising at least one of an accuracy metric, a completion time metric, and a complexity metric.
27. The non-transitory, computer readable medium of claim 25 , further comprising code to update a bias by applying a vector m to compensate for a mean shift in an output activation of the at least one compressed layer and an output activation of the at least one layer.
28. The non-transitory, computer readable medium of claim 25 , further comprising code to determine the low-rank approximation without fine tuning.
29. The non-transitory, computer readable medium of claim 25 , further comprising code to determine the low-rank approximation using singular value decomposition.
30. The non-transitory, computer readable medium of claim 25 , further comprising code to:
determine the set of candidate rank vectors, wherein the set of candidate rank vectors satisfy each target of the set of targets;
evaluate each candidate rank vector of the set of candidate rank vectors; and
select the rank vector of the set of candidate rank vectors based on evaluations of a minimization function using the set of candidate rank vectors.