Systems and methods for modification of neural networks based on estimated edge utility
The present disclosure provides systems and methods for modification (e.g., pruning, compression, quantization, etc.) of artificial neural networks based on estimations of the utility of network connections (also known as “edges”). In particular, the present disclosure provides novel techniques for estimating the utility of one or more edges of a neural network in a fashion that requires far less expenditure of resources than calculation of the actual utility. Based on these estimated edge utilities, a computing system can make intelligent decisions regarding network pruning, network quantization, or other modifications to a neural network. In particular, these modifications can reduce resource requirements associated with the neural network. By making these decisions with knowledge of and based on the utility of various edges, this reduction in resource requirements can be achieved with only a minimal, if any, degradation of network performance (e.g., prediction accuracy).
1. A computer-implemented method of reducing a memory footprint of a machine-learned neural network, the method comprising:
obtaining, by a computing system comprising one or more processors, data descriptive of an initial plurality of weights of the machine-learned neural network, wherein a performance of the machine-learned neural network is characterized by a loss function;
for each respective candidate quantization scheme of a plurality of different candidate quantization schemes, determining, by the computing system, sensitivity values for the initial plurality of weights, wherein the respective candidate quantization scheme is configured to change values of the initial plurality of weights other than by setting all the initial plurality of weights to be zero, wherein determining the sensitivity values comprises, for each respective weight of the initial plurality of weights:
determining a change in the respective weight value based on an initial value of the respective weight and a quantized value of the respective weight;
determining a respective change in a loss by multiplying the change in the respective weight value by a derivative of the loss function with respect to the initial value of the respective weight; and
determining a respective sensitivity value for the respective weight by dividing the respective change in the loss by a magnitude of the initial value of the respective weight;
selecting, by the computing system, one of the candidate quantization schemes based at least in part on the sensitivity values determined for the selected candidate quantization scheme and a cumulative sensitivity value determined for the selected candidate quantization scheme; and
generating, by the computing system, a second plurality of weights by applying the selected one of the candidate quantization schemes to the initial plurality of weights, wherein the second plurality of weights provide a reduced-size parameterization of the machine-learned neural network.
2. The method of claim 1 , wherein a set of weights associated with the reduced-size parameterization of the machine-learned neural network comprises a set of quantized weights and a set of un-quantized weights, wherein the set of quantized weights comprises the second plurality of weights.
3. The method of claim 2 , comprising:
retraining, by the computing system, the set of un-quantized weights while maintaining the set of quantized weights fixed.
4. The method of claim 3 , comprising:
for each respective candidate quantization scheme of the plurality of different candidate quantization schemes, determining, by the computing system, sensitivity values for the retrained set of un-quantized weights, wherein determining the sensitivity values for the retrained set of un-quantized weights comprises, for each respective retrained weight of the retrained set of un-quantized weights:
determining a change in the respective retrained weight value based on an initial value of the respective retrained weight and a quantized value of the respective retrained weight;
determining a respective change in a loss by multiplying the change in the respective retrained weight value by a derivative of the loss function with respect to the initial value of the respective retrained weight; and
determining a respective sensitivity value for the respective retrained weight by dividing the respective change in the loss by a magnitude of the initial value of the respective retrained weight;
selecting, by the computing system, one of the candidate quantization schemes for quantizing the retrained set of un-quantized weights based at least in part on the sensitivity values determined for the selected candidate quantization scheme for quantizing the retrained set of un-quantized weights; and
generating, by the computing system, a third plurality of weights by applying the selected one of the candidate quantization schemes to the retrained set of un-quantized weights, wherein the third plurality of weights provide a further reduced-size parameterization of the machine-learned neural network.
5. The method of claim 4 , comprising:
iteratively performing, by the computing system, retraining and quantizing of the machine-learned neural network until satisfying a criterion based on a rate distortion curve.
6. A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable to cause the computing system to perform operations for reducing a memory footprint of a machine-learned neural network, the operations comprising:
obtaining data descriptive of an initial plurality of weights of the machine-learned neural network, wherein a performance of the machine-learned neural network is characterized by a loss function;
for each respective candidate quantization scheme of a plurality of different candidate quantization schemes, determining sensitivity values for the initial plurality of weights, wherein the respective candidate quantization scheme is configured to change values of the initial plurality of weights other than by setting all the initial plurality of weights to be zero, wherein determining the sensitivity values comprises, for each respective weight of the initial plurality of weights:
determining a change in the respective weight value based on an initial value of the respective weight and a quantized value of the respective weight;
determining a respective change in a loss by multiplying the change in the respective weight value by a derivative of the loss function with respect to the initial value of the respective weight; and
determining a respective sensitivity value for the respective weight by dividing the respective change in the loss by a magnitude of the initial value of the respective weight;
selecting one of the candidate quantization schemes based at least in part on the sensitivity values determined for the selected candidate quantization scheme and a cumulative sensitivity value determined for the selected candidate quantization scheme; and
generating a second plurality of weights by applying the selected one of the candidate quantization schemes to the initial plurality of weights, wherein the second plurality of weights provide a reduced-size parameterization of the machine-learned neural network.
7. The system of claim 6 , wherein a set of weights associated with the reduced-size parameterization of the machine-learned neural network comprises a set of quantized weights and a set of un-quantized weights, wherein the set of quantized weights comprises the second plurality of weights.
8. The system of claim 7 , wherein the operations comprise:
retraining the set of un-quantized weights while maintaining the set of quantized weights fixed.
9. The system of claim 8 , wherein the operations comprise:
for each respective candidate quantization scheme of the plurality of different candidate quantization schemes, determining sensitivity values for the retrained set of un-quantized weights, wherein determining the sensitivity values for the retrained set of un-quantized weights comprises, for each respective retrained weight of the retrained set of un-quantized weights:
determining a change in the respective retrained weight value based on an initial value of the respective retrained weight and a quantized value of the respective retrained weight;
determining a respective change in a loss by multiplying the change in the respective retrained weight value by a derivative of the loss function with respect to the initial value of the respective retrained weight; and
determining a respective sensitivity value for the respective retrained weight by dividing the respective change in the loss by a magnitude of the initial value of the respective retrained weight;
selecting one of the candidate quantization schemes for quantizing the retrained set of un-quantized weights based at least in part on the sensitivity values determined for the selected candidate quantization scheme for quantizing the retrained set of un-quantized weights; and
generating a third plurality of weights by applying the selected one of the candidate quantization schemes to the retrained set of un-quantized weights, wherein the third plurality of weights provide a further reduced-size parameterization of the machine-learned neural network.
10. The system of claim 9 , wherein the operations comprise:
iteratively retraining and quantizing the machine-learned neural network until satisfying a criterion based on a rate distortion curve.
11. One or more non-transitory computer-readable media storing instructions that are executable to cause a computing system to perform operations for reducing a memory footprint of a machine-learned neural network, the operations comprising:
obtaining data descriptive of an initial plurality of weights of the machine-learned neural network, wherein a performance of the machine-learned neural network is characterized by a loss function;
for each respective candidate quantization scheme of a plurality of different candidate quantization schemes, determining sensitivity values for the initial plurality of weights, wherein the respective candidate quantization scheme is configured to change values of the initial plurality of weights other than by setting all the initial plurality of weights to be zero, wherein determining the sensitivity values comprises, for each respective weight of the initial plurality of weights:
determining a change in the respective weight value based on an initial value of the respective weight and a quantized value of the respective weight;
determining a respective change in a loss by multiplying the change in the respective weight value by a derivative of the loss function with respect to the initial value of the respective weight; and
determining a respective sensitivity value for the respective weight by dividing the respective change in the loss by a magnitude of the initial value of the respective weight;
selecting one of the candidate quantization schemes based at least in part on the sensitivity values determined for the selected candidate quantization scheme and a cumulative sensitivity value determined for the selected candidate quantization scheme; and
generating a second plurality of weights by applying the selected one of the candidate quantization schemes to the initial plurality of weights, wherein the second plurality of weights provide a reduced-size parameterization of the machine-learned neural network.
12. The one or more non-transitory computer-readable media of claim 11 , wherein a set of weights associated with the reduced-size parameterization of the machine-learned neural network comprises a set of quantized weights and a set of un-quantized weights, wherein the set of quantized weights comprises the second plurality of weights.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the operations comprise:
retraining the set of un-quantized weights while maintaining the set of quantized weights fixed.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the operations comprise:
for each respective candidate quantization scheme of the plurality of different candidate quantization schemes, determining sensitivity values for the retrained set of un-quantized weights, wherein determining the sensitivity values for the retrained set of un-quantized weights comprises, for each respective retrained weight of the retrained set of un-quantized weights:
determining a change in the respective retrained weight value based on an initial value of the respective retrained weight and a quantized value of the respective retrained weight;
determining a respective change in a loss by multiplying the change in the respective retrained weight value by a derivative of the loss function with respect to the initial value of the respective retrained weight; and
determining a respective sensitivity value for the respective retrained weight by dividing the respective change in the loss by a magnitude of the initial value of the respective retrained weight;
selecting one of the candidate quantization schemes for quantizing the retrained set of un-quantized weights based at least in part on the sensitivity values determined for the selected candidate quantization scheme for quantizing the retrained set of un-quantized weights; and
generating a third plurality of weights by applying the selected one of the candidate quantization schemes to the retrained set of un-quantized weights, wherein the third plurality of weights provide a further reduced-size parameterization of the machine-learned neural network.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the operations comprise:
iteratively retraining and quantizing the machine-learned neural network until satisfying a criterion based on a rate distortion curve.