Efficient Computation for Bayesian Optimization
Systems and methods implement a modular computing environment for Bayesian optimization, decoupling steps of Bayesian optimization across multiple modules; minimizing inter-module dependency; extending functionality of each module; and reusing computing resources and intermediate results within each module. Variable hyperparameterization may reduce computational costs of optimization iterations, while also averting overfitting and destabilization of the Gaussian kernel based on sparser observations of the objective function. Computational complexity of updating the Gaussian kernel may be reduced from the cube to the square of the set of sampled outputs, by deferring computing updates to each hyperparameter while the optimization iterations are ongoing. Furthermore, repeated allocation and release of memory, repeated writing of data in memory to non-volatile storage, and repeated reading of data in non-volatile storage to memory across multiple optimization iterations may be averted, thereby alleviating multiple categories of computing resources, including processing power, memory, storage, from excess performance load.
1 . A method comprising:
pre-allocating, by a computing system, working memory; and
performing, by the computing system, a plurality of iterations of the following steps within the working memory:
optimizing, by the computing system, an acquisition function based on a distribution;
sampling, by the computing system, an output of an objective function; and
updating, by the computing system, a Gaussian kernel of the distribution by regression.
2 . The method of claim 1 , wherein the computing system optimizes the acquisition function by performing a gradient descent computation over the distribution.
3 . The method of claim 1 , wherein the computing system optimizes the acquisition function by performing global and local searches over the distribution.
4 . The method of claim 1 , wherein the computing system optimizes the acquisition function by iteratively searching linear approximations of the distribution.
5 . The method of claim 1 , wherein the computing system updates the kernel of the distribution by performing variable hyperparameterization.
6 . The method of claim 1 , wherein the computing system updates the kernel of the distribution by incremental updates.
7 . The method of claim 1 , wherein the computing system updates the kernel of the distribution by sub-sampling sampled outputs of the objective function.
8 . A system comprising:
one or more processors; and
memory communicatively coupled to the one or more processors, the memory storing computer-executable modules executable by the one or more processors that, when executed by the one or more processors, perform associated operations, the computer-executable modules comprising:
a memory pre-allocation module configuring the one or more processors to pre-allocate working memory; and
a nonlinear optimization module, a sampling module, and a Gaussian Process module, respectively configuring the one or more processors to perform a plurality of iterations of the following steps within the working memory:
optimize an acquisition function based on a distribution;
sample an output of an objective function; and
update a kernel of the distribution by regression.
9 . The system of claim 8 , wherein the nonlinear optimizing module further comprises a gradient descent submodule configuring the one or more processors to optimize the acquisition function by performing a gradient descent computation.
10 . The system of claim 8 , wherein the nonlinear optimizing module further comprises a search submodule configuring the one or more processors to optimize the acquisition function by performing global and local searches over the distribution.
11 . The system of claim 8 , wherein the nonlinear optimizing module further comprises an iterative search submodule configuring the one or more processors to optimize the acquisition function by iteratively searching linear approximations of the distribution.
12 . The system of claim 8 , wherein the Gaussian Process module further comprises a variable hyperparameterization submodule configuring the one or more processors to update the kernel of the distribution by performing variable hyperparameterization.
13 . The system of claim 8 , wherein the Gaussian Process module further comprises an incremental updating submodule configuring the one or more processors to update the kernel of the distribution by incremental updates.
14 . The system of claim 8 , wherein the Gaussian Process module further comprises a sub-sampling submodule configuring the one or more processors to update the kernel of the distribution by sub-sampling sampled outputs of the objective function.
15 . A computer-readable storage medium storing computer-readable instructions executable by one or more processors, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
pre-allocating, by a computing system, working memory; and
performing, by the computing system, a plurality of iterations of the following steps within the working memory:
optimizing, by the computing system, an acquisition function based on a distribution;
sampling, by the computing system, an output of an objective function; and
updating, by the computing system, a Gaussian kernel of the distribution by regression.
16 . The computer-readable storage medium of claim 15 , wherein the computing system optimizes the acquisition function by performing a gradient descent computation over the distribution.
17 . The computer-readable storage medium of claim 15 , wherein the computing system optimizes the acquisition function by performing global and local searches over the distribution.
18 . The computer-readable storage medium of claim 15 , wherein the computing system optimizes the acquisition function by iteratively searching linear approximations of the distribution.
19 . The computer-readable storage medium of claim 15 , wherein the computing system updates the kernel of the distribution by performing variable hyperparameterization.
20 . The computer-readable storage medium of claim 15 , wherein the computing system updates the kernel of the distribution by sub-sampling sampled outputs of the objective function.