Robust and data-efficient blackbox optimization
The present disclosure provides iterative blackbox optimization techniques that estimate the gradient of a function. According to an aspect of the present disclosure, a plurality of perturbations used at each iteration can be sampled from a non-orthogonal sampling distribution. As one example, in some implementations, perturbations that have been previously evaluated in previous iterations can be re-used at the current iteration, thereby conserving computing resources because the re-used perturbations do not need to be re-evaluated at the current iteration. In another example, in addition or alternatively to the use of previously evaluated perturbations, the perturbations evaluated at the current iteration can be sampled from a non-orthogonal sampling distribution.
1 . A computer-implemented method, comprising:
obtaining, by one or more computing devices, data descriptive of current values of a plurality of parameters of a machine-learned model; and
for at least one of one or more iterations:
identifying, by the one or more computing devices, one or more previously evaluated perturbations based at least in part on the current values of the plurality of parameters of the machine-learned model;
sampling, by the one or more computing devices, a plurality of perturbations to the current values of the plurality of parameters of the machine-learned model from a non-orthogonal sampling distribution, the plurality of perturbations including the one or more previously evaluated perturbations;
determining, by the one or more computing devices, a plurality of performance values respectively for the plurality of perturbations, wherein the performance value for each perturbation is generated through evaluation, by a performance evaluation function, of a performance of the machine-learned model with the current values of its parameters perturbed according to the perturbation, wherein determining the plurality of performance values comprises re-using one or more previously evaluated performance values respectively for the one or more previously evaluated perturbations;
performing, by the one or more computing devices, a regression with respect to the plurality of perturbations and the plurality of performance values to estimate a gradient of the performance evaluation function; and
modifying, by the one or more computing devices, the current value of at least one of the plurality of parameters of the machine-learned model based at least in part on the estimated gradient of the performance evaluation function; and
after the one or more iterations, providing, by the one or more computing devices, final values of the plurality of parameters of the machine-learned model as an output,
wherein the one or more previously evaluated perturbations are included within a trust region associated with the current values of the plurality of parameters, and
wherein identifying, by the one or more computing devices, the one or more previously evaluated perturbations that are included within the trust region comprises identifying, by the one or more computing devices, any previously evaluated perturbations that are within a radius from the current values of the plurality of parameters.
2 . The computer-implemented method of claim 1 , wherein identifying, by the one or more computing devices, the one or more previously evaluated perturbations that are included within the trust region comprises identifying, by the one or more computing devices, a fixed fraction of previously evaluated perturbations that are closest to the current values of the plurality of parameters.
3 . The computer-implemented method of claim 1 , wherein performing, by the one or more computing devices, the regression with respect to the plurality of perturbations and the plurality of performance values comprises determining, by the one or more computing devices, a forward finite-difference evolution strategy estimator based on the plurality of perturbations and the plurality of performance values.
4 . The computer-implemented method of claim 1 , wherein performing, by the one or more computing devices, the regression with respect to the plurality of perturbations and the plurality of performance values comprises determining, by the one or more computing devices, an antithetic evolution strategy estimator based on the plurality of perturbations and the plurality of performance values.
5 . The computer-implemented method of claim 1 , wherein the machine-learned model comprises a reinforcement learning policy and the performance evaluation function comprises a reward function that determines a reward for actions taken in accordance with the reinforcement learning policy.
6 . The computer-implemented method of claim 1 , wherein the machine-learned model comprises a neural network.
7 . The computer-implemented method of claim 1 , wherein the parameters of the machine-learned model comprise hyperparameters of the machine-learned model.
8 . The computer-implemented method of claim 1 , wherein the machine-learned model comprises a structured network with weight sharing mechanisms.
9 . The computer-implemented method of claim 1 , wherein performing, by the one or more computing devices, the regression with respect to the plurality of perturbations and the plurality of performance values comprises performing, by the one or more computing devices, an under-constrained linear regression with respect to the plurality of perturbations and the plurality of performance values.
10 . The computer-implemented method of claim 1 , wherein performing, by the one or more computing devices, the regression with respect to the plurality of perturbations and the plurality of performance values comprises performing, by the one or more computing devices, an L1- or L2-regularized regression with respect to the plurality of perturbations and the plurality of performance values.
11 . A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining data descriptive of current values of a plurality of parameters of a machine-learned model; and
for at least one iteration of one or more iterations:
identifying one or more previously evaluated perturbations that are included within a trust region associated with the current values of the plurality of parameters;
accessing one or more previously evaluated performance values respectively for the one or more previously evaluated perturbations that are included within the trust region;
sampling a plurality of additional perturbations to the current values of the plurality of parameters of the machine-learned model from a sampling distribution;
determining a plurality of additional performance values respectively for the plurality of additional perturbations, wherein the performance value for each additional perturbation is generated through evaluation, by a performance evaluation function, of a performance of the machine-learned model with the current values of its parameters perturbed according to the additional perturbation;
performing a regression with respect to a first combination of the one or more previously evaluated perturbations with the plurality of additional perturbations and a second combination of the one or more previously evaluated performance values with the plurality of additional performance values to estimate a gradient of the performance evaluation function; and
modifying the current value of at least one of the plurality of parameters of the machine-learned model based at least in part on the estimated gradient of the performance evaluation function,
wherein identifying the one or more previously evaluated perturbations that are included within the trust region comprises identifying a fixed fraction of previously evaluated perturbations that are closest to the current values of the plurality of parameters.
12 . The computing system of claim 11 , wherein identifying the one or more previously evaluated perturbations that are included within the trust region comprises identifying any previously evaluated perturbations that are within a radius from the current values of the plurality of parameters.
13 . The computing system of claim 11 , wherein performing the regression comprises determining a forward finite-difference evolution strategy estimator based on the first combination of the one or more previously evaluated perturbations with the plurality of additional perturbations and the second combination of the one or more previously evaluated performance values with the plurality of additional performance values.
14 . The computing system of claim 11 , wherein performing the regression comprises determining an antithetic evolution strategy estimator based on the first combination of the one or more previously evaluated perturbations with the plurality of additional perturbations and the second combination of the one or more previously evaluated performance values with the plurality of additional performance values.
15 . One or more non-transitory computer-readable media that collectively store operations that when executed by a computing system cause the computing system to perform operations, the operations comprising:
obtaining data descriptive of current values of a plurality of parameters of a machine-learned model; and
for at least one of one or more iterations:
identifying, by the one or more computing devices, one or more previously evaluated perturbations based at least in part on the current values of the plurality of parameters of the machine-learned model;
sampling a plurality of perturbations to the current values of the plurality of parameters of the machine-learned model from a non-orthogonal sampling distribution, the plurality of perturbations including the one or more previously evaluated perturbations;
determining a plurality of performance values respectively for the plurality of perturbations, wherein the performance value for each perturbation is generated through evaluation, by a performance evaluation function, of a performance of the machine-learned model with the current values of its parameters perturbed according to the perturbation, wherein determining the plurality of performance values comprises re-using one or more previously evaluated performance values respectively for the one or more previously evaluated perturbations;
performing a regression with respect to the plurality of perturbations and the plurality of performance values to estimate a gradient of the performance evaluation function; and
modifying the current value of at least one of the plurality of parameters of the machine-learned model based at least in part on the estimated gradient of the performance evaluation function; and
after the one or more iterations, providing final values of the plurality of parameters of the machine-learned model as an output,
wherein the one or more previously evaluated perturbations are included within a trust region associated with the current values of the plurality of parameters, and
wherein identifying the one or more previously evaluated perturbations that are included within the trust region comprises identifying any previously evaluated perturbations that are within a radius from the current values of the plurality of parameters.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein identifying the one or more previously evaluated perturbations that are included within the trust region comprises identifying a fixed fraction of previously evaluated perturbations that are closest to the current values of the plurality of parameters.
17 . The one or more non-transitory computer-readable media of claim 15 , wherein performing the regression with respect to the plurality of perturbations and the plurality of performance values comprises determining a forward finite-difference evolution strategy estimator based on the plurality of perturbations and the plurality of performance values.
18 . The one or more non-transitory computer-readable media of claim 15 , plurality of perturbations and the plurality of performance values comprises determining an antithetic evolution strategy estimator based on the plurality of perturbations and the plurality of performance values.
19 . The one or more non-transitory computer-readable media of claim 15 , wherein the machine-learned model comprises a reinforcement learning policy and the performance evaluation function comprises a reward function that determines a reward for actions taken in accordance with the reinforcement learning policy.
20 . The one or more non-transitory computer-readable media of claim 15 , wherein the machine-learned model comprises a neural network.