GENETIC ALGORITHM-BASED ADAPTIVE BATCH SELECTION FOR HESSIAN QUANTIZATION IN NEURAL NETWORKS
In aspect, a computerized method of a genetic algorithm-based adaptive batch selection for hessian quantization in neural networks comprising: with at least one computer processer, computing a Hessian Matrix; performing an Eigenvalue Analysis on the Hessian matrix to generate a Hessian matrix eigenvalue that provides information about the curvature of the loss surface; determining a quantization level based on the Hessian matrix eigenvalue; using the quantization Level to set an appropriate quantization level for a layer weights of a neural network; and applying the quantization level to the layer weights of the neural network. This involves mapping the continuous floating-point values of the weights to discrete levels based on the determined quantization intervals.
1 . A computerized method of a genetic algorithm-based adaptive batch selection for hessian quantization in neural networks comprising:
with at least one computer processer, computing a Hessian Matrix;
performing an Eigenvalue Analysis on the Hessian matrix to generate a Hessian matrix eigenvalue that provides information about the curvature of the loss surface;
determining a quantization level based on the Hessian matrix eigenvalue;
using the quantization Level to set an appropriate quantization level for a layer weights of a neural network; and
applying the quantization level to the layer weights of the neural network. This involves mapping the continuous floating-point values of the weights to discrete levels based on the determined quantization intervals.
2 . The method of claim 1 , wherein Hessian matrix provides information about a curvature of a loss surface with respect to a plurality of model parameters of at least one neural network implemented in a computing system.
3 . The computerized method of claim 2 , wherein the computing of the Hessian matrix comprises:
calculating second-order partial derivatives of a loss function with respect to each parameter.
4 . The computerized method of claim 3 , wherein the Hessian matrix is computed with an analytical algorithm.
5 . The computerized method of claim 3 , wherein the Hessian matrix is approximated numerical algorithm.
6 . The computerized method of claim 2 , wherein a higher eigenvalue indicates a region of high curvature.
7 . The computerized method of claim 6 , wherein a lower eigenvalue indicates a region of low curvature.
8 . The computerized method of claim 7 , wherein the regions of high curvature is indicated.
9 . The computerized method of claim 8 further comprising:
setting a finer quantization level for the layer weight of the neural network to preserve a model accuracy.
10 . The computerized method of claim 7 , wherein a region of low curvature is detected.
11 . The computerized method of claim 10 further comprising:
setting a coarser quantization level for the layer weight of the neural network to preserve a model accuracy.
12 . The computerized method of claim 7 , wherein the step of applying the quantization level to the layer weights of the neural network further comprises:
mapping a continuous floating-point value of a plurality of weights to a plurality of discrete levels based on the quantization intervals.
13 . The computerized method of claim 12 further comprising:
using a genetic algorithm to generate an optimal batch combination out of a solution space.
14 . The computerized method of claim 13 further comprising:
implementing an application of genetic algorithm specifically designed for a batch selection in the context of the Hessian Quantization.
15 . The computerized method of claim 14 , wherein the optimization of the batch selection based on a requirements of each layer of the neural network.
16 . The computerized method of claim 15 further comprising:
enabling the genetic algorithm to dynamically adapt and optimize the batch selection.
17 . The computerized method of claim 16 further comprising:
providing a plurality of interconnections between the genetic algorithm and the Hessian quantization process.
18 . The computerized method of claim 17 , wherein the genetic algorithm interacts with the quantization process to determine a most optimal batch combination for each layer of the neural network.