Determining control policies for robots with noise-tolerant structured exploration
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for optimizing the determination of control policies for robots through the performance of simulations of robots and real-world context to determine control policy parameters.
1. A method for determining a control policy for a robot, the method comprising:
receiving a specification in the form of an objective function ƒ of a task to be performed by the robot,
wherein ƒ is a scalar-valued objective function of a control policy, and the control policy determines, for each state s of the robot, an action u to be performed by the robot, and
wherein the objective function ƒ represents how well the robot performs the task;
determining the control policy for the robot to perform the task by simulating operation of the robot to determine the control policy x* by solving an optimization problem of the form
x
*
=
argmin
x
∈
ℝ
n
f
(
x
)
,
wherein x∈ n is a state of the control policy encountered during simulating the operation of the robot, and wherein solving the optimization problem comprises estimating gradients of the objective function ƒ using a finite difference procedure to estimate the gradients in perturbation directions defined by rows of a balanced spinner, the balanced spinner being a two-dimensional matrix such that the absolute value of an inner product between any one of multiple rows of the two-dimensional matrix and any one of multiple columns of the two-dimensional matrix is bounded by a threshold value, and
causing the robot to operate under control of the determined control policy x*.
2. The method of claim 1 , wherein the control policy for the robot is a parameterized control policy and the x is a vectorized version of the parameters of the control policy.
3. The method of claim 1 , wherein the control policy for the robot is represented by weights and biases of a neural network model, the method further comprising:
training the neural network model to minimize the objective function ƒ to learn the control policy x*; and
transferring the learned control policy x* to the robot.
4. The method of claim 1 ,
wherein determining the control policy for the robot comprises performing reinforcement learning policy optimization to learn the control policy x*,
wherein ƒ is a negative of an expected return over the simulation, and
wherein the method further comprises:
transferring the learned control policy x* to the robot.
5. The method of claim 1 , wherein the optimization problem is of the form
x
*
=
argmin
x
∈
Ω
ℝ
n
f
(
x
)
,
wherein each x and x* is constrained to be in a subset Ω of n .
6. The method claim 1 , wherein, in solving the optimization problem, evaluations of ƒ at a state x return ƒ(x)+ε(x), where ε(x) is a noise term.
7. The method of claim 1 , wherein the balanced spinners are (α,β)-balanced spinners, and wherein 0<α≤1 and 0<β≤1.
8. The method of claim 7 , wherein β≈1 and the rows of the structured matrix are nearly orthogonal.
9. The method of claim 8 , wherein α and β each differ from 1 by a respective amount that is not more than about
k
n
,
wherein k≈1 is a constant and n is the dimensionality of x, whereby the rows of the structured matrix are substantially.
10. The method of claim 7 , wherein α=1 and β=1.
11. The method of claim 10 , wherein the structured matrix is of the form
(
Q
p
x
)
T
,
a quadratic residue matrix.
12. The method of claim 10 , wherein the structured matrix is a Hadamard matrix.
13. The method of claim 10 , wherein the structured matrix is a (1,1)-balanced spinner of the form MD, wherein M is a (1,1)-balanced spinner and D is a diagonal matrix D∈ n with diagonal entries that are either −1 or +1.
14. The method of claim 10 , wherein the structured matrix is a Hadamard matrix, a Hadamard matrix right multiplied by a diagonal matrix with diagonal entries that are either −1 or +1, or a quadratic residue matrix.
15. The method of claim 13 , wherein the diagonal entries of the diagonal matrix are taken independently at random from the set {−1, +1}.
16. The method of claim 15 , comprising:
estimating the gradients of the objective function ƒ by fast matrix-vector multiplication of the structured matrix using an FWHT (Fast Walsh-Hadamard Transform) or an FFT (Fast Fourier Transform).
17. A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving a specification in the form of an objective function ƒ of a task to be performed by the robot,
wherein ƒ is a scalar-valued objective function of a control policy, and the control policy determines, for each state s of the robot, an action u to be performed by the robot, and
wherein the objective function ƒ represents how well the robot performs the task;
determining the control policy for the robot to perform the task by simulating operation of the robot to determine the control policy x* by solving an optimization problem of the form
x
*
=
argmin
x
∈
ℝ
n
f
(
x
)
,
wherein x∈ n is a state of the control policy encountered during simulating the operation of the robot, and wherein solving the optimization problem comprises estimating gradients of the objective function ƒ using a finite difference procedure to estimate the gradients in perturbation directions defined by rows of a balanced spinner, the balanced spinner being a two-dimensional matrix such that the absolute value of an inner product between any one of multiple rows of the two-dimensional matrix and any one of multiple columns of the two-dimensional matrix is bounded by a threshold value, and
causing the robot to operate under control of the determined control policy x*.
18. The system of claim 17 , wherein the control policy for the robot is a parameterized control policy and the x is a vectorized version of the parameters of the control policy.
19. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving a specification in the form of an objective function ƒ of a task to be performed by the robot,
wherein ƒ is a scalar-valued objective function of a control policy, and the control policy determines, for each state s of the robot, an action u to be performed by the robot, and
wherein the objective function ƒ represents how well the robot performs the task; and
determining the control policy for the robot to perform the task by simulating operation of the robot to determine the control policy x* by solving an optimization problem of the form
x
*
=
argmin
x
∈
ℝ
n
f
(
x
)
,
wherein x∈ n is a state of the control policy encountered during simulating the operation of the robot, and wherein solving the optimization problem comprises estimating gradients of the objective function ƒ using a finite difference procedure to estimate the gradients in perturbation directions defined by rows of a balanced spinner, the balanced spinner being a two-dimensional matrix such that the absolute value of an inner product between any one of multiple rows of the two-dimensional matrix and any one of columns of the two-dimensional matrix is bounded by a threshold value, and
causing the robot to operate under control of the determined control policy x*.
20. The computer-readable storage media of claim 19 , wherein the control policy for the robot is a parameterized control policy and the x is a vectorized version of the parameters of the control policy.