Systems and methods for applying semi-discrete calculus to meta machine learning
A method and system for building and implementing a meta-machine learning (meta-ML) optimization engine for a neural network (NN) or a machine learning (ML) connective model. A computer processor may iteratively simulate a backpropagation algorithm by executing a sequence of optimization steps. At each optimization step a position of a loss function may be determined that may be closer than a previously determined position of the loss function to a local minimum. A computer processor may compute and store after each iteration a detachment of the loss function, learning rate, and optimal learning rate. A computer processor may train a machine learning connective model to model the optimal learning rates of the simulated backpropagation algorithm. The meta-ML optimization engine may be implemented for a NN or ML connective model by generating a modified backpropagation algorithm in which algorithmic features of gradient descent may be replaced by the meta-ML optimization engine.
1 . A method for building a meta-machine learning (meta-ML) optimization engine for a neural network (NN) or a machine learning (ML) connective model, the method comprising using a computer processor:
computing local minima of a loss function for a data set;
storing a detachment of the loss function at a selected position of a parameter of the loss function;
iteratively simulating a backpropagation algorithm by executing a sequence of optimization steps, wherein at each iteration an optimization step is configured to determine a position of a parameter of the loss function that is closer than the selected position of the parameter of the loss function or a position of the parameter of the loss function determined in a previous iteration to a local minimum of the loss function, wherein the simulated backpropagation algorithm avoids calculating a derivative of the loss function;
computing at each iteration a detachment of the loss function and learning rate at the position of a determined parameter of the loss function;
computing at each iteration one or more optimal learning rates, wherein an optimal learning rate comprises an absolute value of a difference between the position of a determined parameter of the loss function and the position of a local minimum of the loss function, wherein the local minimum of the loss function is determined by an analysis of detachments;
storing after each iteration the computed detachment, learning rate, and optimal learning rates;
using one or more of the stored detachments as an explaining variable in the simulated backpropagation algorithm; and
training a ML connective model to model the optimal learning rates of the simulated backpropagation algorithm, wherein the training is initiated before a termination of the simulated backpropagation algorithm, wherein the training is based on data accumulated in one or more iterations of the simulated backpropagation algorithm, and wherein the training comprises incrementally updating a learning rate of the ML connective model based on batches of the stored learning rates from the simulated backpropagation algorithm.
2 . The method of claim 1 comprising: computing at each iteration an absolute value of the difference between the position of a determined parameter of the loss function and the position of a determined parameter of the loss function in a previous iteration.
3 . The method of claim 2 comprising: iteratively simulating the backpropagation algorithm until the absolute value of the difference between the position of a determined parameter of the loss function and the position of the determined parameter of the loss function in a previous iteration satisfies a minimum threshold.
4 . The method of claim 1 wherein the backpropagation algorithm is a variant of RProp that uses a detachment operator in place of a derivative operator.
5 . The method of claim 1 wherein the selected position of the parameter of the loss function is selected randomly.
6 . The method of claim 1 wherein the backpropagation algorithm is iteratively simulated by a Monte Carlo method.
7 . The method of claim 1 wherein the ML connective model is XGBoost.
8 . The method of claim 1 comprising: collecting into respective features of the ML connective model tuples of consecutive identical detachments,
wherein if a dimensionality of the tuple is equal to or greater than a threshold an explained variable of the ML connective model is explored; and
wherein if the dimensionality of the tuple is less than the threshold the explained variable of the ML connective model is exploited.
9 . A method for implementing a meta-machine learning (meta-ML) optimization engine for a neural network (NN) or a machine learning (ML) connective model, the method comprising using a computer processor:
generating training data and data by preparing raw training data and raw data;
generating a modified backpropagation algorithm by replacing algorithmic traits of gradient descent in a backpropagation algorithm that uses gradient descent with the meta-ML optimization engine, wherein the modified backpropagation algorithm comprises:
computing at each iteration a detachment of a loss function and learning rate at a position of a determined parameter of the loss function a plurality of iterations, wherein the modified backpropagation algorithm avoids calculating a derivative of the loss function; and
computing at each iteration one or more optimal learning rates, wherein an optimal learning rate comprises an absolute value of a difference between the position of a determined parameter of the loss function and the position of a local minimum of the loss function, wherein the local minimum of the loss function is determined by an analysis of detachments;
wherein one or more of the detachments are used as an explaining variable in the modified backpropagation algorithm;
training the NN or ML connective model using training data and the modified backpropagation algorithm, wherein the training is initiated before a termination of the modified backpropagation algorithm, wherein the training is based on data accumulated in one or more iterations of the modified backpropagation algorithm, and wherein the training comprises incrementally updating a learning rate of the NN or ML connective model based on batches of the computed learning rates from the modified backpropagation algorithm; and
applying the trained NN or ML connective model to data.
10 . The method of claim 9 wherein the algorithmic traits of gradient descent only use numerical differentiation.
11 . The method of claim 9 comprising: training the NN or ML connective model to detect financial fraud.
12 . A system for building a meta-machine learning (meta-ML) optimization engine for a neural network (NN) or a machine learning (ML) connective model, the system comprising:
a memory; and
one or more processors configured to:
compute local minima of a loss function for a data set;
store a detachment of the loss function at a selected position of a parameter of the loss function;
iteratively simulate a backpropagation algorithm by executing a sequence of optimization steps, wherein at each iteration an optimization step is configured to determine a position of a parameter of the loss function that is closer than the selected position of the parameter of the loss function or a position of the parameter of the loss function determined in a previous iteration to a local minimum of the loss function, wherein the simulated backpropagation algorithm avoids calculating a derivative of the loss function;
compute at each iteration a detachment of the loss function and learning rate at the position of a determined parameter of the loss function;
compute at each iteration one or more optimal learning rates, wherein an optimal learning rate comprises an absolute value of a difference between the position of a determined parameter of the loss function and the position of a local minimum of the loss function, wherein the local minimum of the loss function is determined by an analysis of detachments;
store after each iteration the computed detachment, learning rate, and optimal learning rates;
use one or more of the stored detachments as an explaining variable in the simulated backpropagation algorithm; and
train a ML connective model to model the optimal learning rates of the simulated backpropagation algorithm, wherein the training is initiated before a termination of the simulated backpropagation algorithm, wherein the training is based on data accumulated in one or more iterations of the simulated backpropagation algorithm, and wherein the training comprises incrementally updating a learning rate of the ML connective model based on batches of the stored learning rates from the simulated backpropagation algorithm.
13 . The system of claim 12 wherein the one or more processors are configured to compute at each iteration an absolute value of the difference between the position of a determined parameter of the loss function and the position of a determined parameter of the loss function in a previous iteration.
14 . The system of claim 13 wherein the one or more processors are configured to iteratively simulate the backpropagation algorithm until the absolute value of the difference between the position of a determined parameter of the loss function and the position of the determined parameter of the loss function in a previous iteration satisfies a minimum threshold.
15 . The system of claim 12 wherein the backpropagation algorithm is a variant of RProp that uses a detachment operator in place of a derivative operator.
16 . The system of claim 12 wherein the selected position of the parameter of the loss function is selected randomly.
17 . The system of claim 12 wherein the one or more processors are configured to iteratively simulate the backpropagation algorithm by using a Monte Carlo method.
18 . The system of claim 12 wherein the ML connective model is XGBoost.
19 . The system of claim 12 wherein the one or more processors are configured to collect into respective features of the ML connective model tuples of consecutive identical detachments,
wherein if a dimensionality of the tuple is greater than or equal to a threshold an explained variable of the ML connective model is explored; and
wherein if the dimensionality of the tuple is less than the threshold the explained variable of the ML connective model is exploited.
20 . A system for implementing a meta-machine learning (meta-ML) optimization engine for a neural network (NN) or a machine learning (ML) connective model, the system comprising:
a memory; and
one or more processors configured to:
generate training data and data by preparing raw training data and raw data;
generate a modified backpropagation algorithm by replacing algorithmic traits of gradient descent in a backpropagation algorithm that uses gradient descent with the meta-ML optimization engine, wherein the modified backpropagation algorithm comprises:
computing at each iteration a detachment of a loss function and learning rate at a position of a determined parameter of the loss function a plurality of iterations, wherein the modified backpropagation algorithm avoids calculating a derivative of the loss function; and
computing at each iteration one or more optimal learning rates, wherein an optimal learning rate comprises an absolute value of a difference between the position of a determined parameter of the loss function and the position of a local minimum of the loss function, wherein the local minimum of the loss function is determined by an analysis of detachments;
wherein one or more of the detachments are used as an explaining variable in the modified backpropagation algorithm;
train the NN or ML connective model using the training data and the modified backpropagation algorithm, wherein the training is initiated before a termination of the modified backpropagation algorithm, wherein the training is based on data accumulated in one or more iterations of the modified backpropagation algorithm, and wherein the training comprises incrementally updating a learning rate of the NN or ML connective model based on batches of the computed learning rates from the modified backpropagation algorithm; and
apply the trained NN or ML connective model to the data.
21 . The system of claim 20 wherein the algorithmic traits of gradient descent only use numerical differentiation.
22 . The system of claim 20 wherein the one or more processors are configured to train the NN or ML connective model to detect financial fraud.