Methods of generating auxilary functions for use with machine learning methods
Certain aspects of the present disclosure provide techniques for optimizing a cost function of a deep learning algorithm. One example method generally includes executing, via a processing apparatus, the machine learning model, wherein the machine learning model can be optimized using gradient descent and evaluating output of the machine learning model using a cost function. The method further involves determining a gradient of the cost function and generating an auxiliary function based on the gradient. The method further involves identifying a root of the auxiliary function corresponding to a minimum point of the cost function, wherein calculation of the root is numerically stable and updating the machine learning model based on the root of the auxiliary function.
1 . A method for training a machine learning model, comprising:
executing, via a processing apparatus comprising a processor and a memory, the machine learning model;
providing an input to the machine learning model based on training data comprising:
the input; and
a ground truth label representing a known correct output;
receiving an output from the machine learning model in response to the input;
evaluating a cost function that compares the output from the machine learning model to the ground truth label representing the known correct output;
determining a gradient of the cost function;
generating an auxiliary function based on an inner product of the gradient, wherein the inner product of the gradient comprises a sum of squares of a plurality of components of the gradient, wherein the gradient has a vector output and the auxiliary function has a scalar output, and wherein the plurality of components of the gradient comprise values corresponding to a plurality of dimensions of a vector;
calculating, using the processor, a root of the auxiliary function corresponding to a minimum point of the cost function, wherein the calculating of the root is numerically stable and suitable for the processor as a result of the scalar output of the auxiliary function; and
updating one or more parameters of the machine learning model using a step size that is determined based on the root of the auxiliary function to produce a trained machine learning model.
2 . The method of claim 1 , wherein training the machine learning model does not require inverting a Jacobian matrix of the gradient to iteratively estimate the minimum point of the cost function.
3 . The method of claim 1 , wherein identifying the root of the auxiliary function comprises applying a modified Newton's method to the auxiliary function.
4 . The method of claim 1 , further comprising:
determining the root of the auxiliary function corresponds with a local maximum or a saddle point of the cost function; and
identifying the minimum point of the cost function using a conventional gradient descent technique.
5 . The method of claim 1 , further comprising, prior to executing the machine learning model, obtaining the training data for the machine learning model.
6 . A system comprising:
a memory including computer executable instructions; and
a processor configured to execute the computer executable instructions and cause the system to perform operations for training a machine learning model using a cost function, the operations comprising:
executing, via the system, the machine learning model;
providing an input to the machine learning model based on training data comprising:
the input; and
a ground truth label representing a known correct output;
receiving an output from the machine learning model in response to the input;
evaluating a cost function that compares the output from the machine learning model to the ground truth label representing the known correct output;
determining a gradient of the cost function;
generating an auxiliary function based on an inner product of the gradient, wherein the inner product of the gradient comprises a sum of squares of a plurality of components of the gradient, wherein the gradient has a vector output and the auxiliary function has a scalar output, and wherein the plurality of components of the gradient comprise values corresponding to a plurality of dimensions of a vector;
calculating, using the processor, a root of the auxiliary function corresponding to a minimum point of the cost function, wherein the calculating of the root is numerically stable and suitable for the processor as a result of the scalar output of the auxiliary function; and
updating one or more parameters of the machine learning model using a step size that is determined based on the root of the auxiliary function to produce a trained machine learning model.
7 . The system of claim 6 , wherein training the machine learning model does not require inverting a Jacobian matrix of the gradient to iteratively estimate the minimum point of the cost function.
8 . The system of claim 6 , wherein identifying the root of the auxiliary function comprises applying a modified Newton's method to the auxiliary function.
9 . The system of claim 6 , the operations further comprising:
determining the root of the auxiliary function corresponds with a local maximum or a saddle point of the cost function; and
identifying the minimum point of the cost function using a conventional gradient descent technique.
10 . The system of claim 6 , the operations further comprising, prior to executing the machine learning model, obtaining the training data for the machine learning model.
11 . A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processor of a computing device, perform operations for training a machine learning model using a cost function, the operations comprising:
executing, via the computing device, the machine learning model;
providing an input to the machine learning model based on training data comprising:
the input; and
a ground truth label representing a known correct output;
receiving an output from the machine learning model in response to the input;
evaluating a cost function that compares the output from the machine learning model to the ground truth label representing the known correct output;
determining a gradient of the cost function;
generating an auxiliary function based on an inner product of the gradient, wherein the inner product of the gradient comprises a sum of squares of a plurality of components of the gradient, wherein the gradient has a vector output and the auxiliary function has a scalar output, and wherein the plurality of components of the gradient comprise values corresponding to a plurality of dimensions of a vector;
calculating, using the processor, a root of the auxiliary function corresponding to a minimum point of the cost function, wherein the calculating of the root is numerically stable and suitable for the processor as a result of the scalar output of the auxiliary function; and
updating one or more parameters of the machine learning model using a step size that is determined based on the root of the auxiliary function to produce a trained machine learning model.
12 . The computer-readable medium of claim 11 , wherein identifying the root of the auxiliary function comprises applying a modified Newton's method to the auxiliary function.
13 . The computer-readable medium of claim 11 , the operations further comprising:
determining the root of the auxiliary function corresponds with a local maximum or a saddle point of the cost function; and
identifying the minimum point of the cost function using a conventional gradient descent technique.