Universal approximation of reinforcement learning solutions with hypernetworks

1 . A method of training a hypernetwork, the method comprising:

initializing the hypernetwork;

sampling a mini-batch of system parameter sets from a plurality of system parameter sets;

generating, using the hypernetwork, policy weights for a policy;

generating, using the hypernetwork, value function weights for a value function;

calculating a first loss, L_pred, using the mini-batch;

calculating a second loss, L_TD, using the mini-batch;

updating the hypernetwork using the first loss and the second loss; and

repeating the sampling through the updating until the hypernetwork has converged.

2 . The method of claim 1 , wherein L_pred is based on predicting a near-optimal action and based on predicting a near-optimal value, and

wherein L_TD is based on moving the predicted near-optimal value toward a current value estimate.

3 . The method of claim 1 , further comprising:

initializing the plurality of system parameter sets to be empty;

using a software model to define a set n-tuples, each n-tuple of the set of n-tuples comprising values of state, action, and reward; and

solving, using an RL solver applied to the set of n-tuples, for the policy and the value function, wherein the policy and the value function form a first set of data, wherein the first set of data comprises an indication of a first context, and the first set of data further comprises a first plurality of system parameters; and

augmenting the plurality of system parameter sets with the first set of data.

4 . The method of claim 1 , further comprising:

receiving a plurality of first trajectories; and

solving for a plurality of first reward parameters and a plurality of first transition dynamics parameters based on the plurality of first trajectories.

5 . The method of claim 4 , wherein the generating, using the hypernetwork, the policy weights for the policy comprises solving for a plurality of first policy parameters based on the plurality of first reward parameters and the plurality of first transition dynamics parameters and based on a plurality of second reward parameters and a plurality of second transition dynamics parameters, and

wherein the generating, using the hypernetwork, the value function weights for the value function comprises solving for a plurality of first value parameters based on the plurality of first reward parameters and the plurality of first transition dynamics parameters and based on the plurality of second reward parameters and the plurality of second transition dynamics parameters.

6 . The method of claim 1 , wherein L_pred comprises an expectation of a first mean square loss and a second mean square loss, wherein the first mean square loss is based on a first difference between an experienced value and a ground truth value and the second mean square loss is based on a second difference between a chosen action and a ground truth action.

7 . The method of claim 1 , where L_TD comprises an expectation of a third mean square loss, wherein the third mean square loss is based on an obtained reward for moving to a next state summed with a difference of a second experienced value with a ground truth value.

8 . The method of claim 5 , further comprising:

receiving a voice command from a user;

mapping the voice command to a context;

mapping the context to a hypernetwork input;

obtaining the policy by inputting the hypernetwork input to the hypernetwork; and

manipulating, using and in response to the voice command, an object with a robot arm.

9 . An apparatus comprising:

one or more processors; and

one or more memories, the one or more memories storing instructions configured to cause the apparatus to at least:

initialize a hypernetwork;

sample a mini-batch of system parameter sets from a plurality of system parameter sets;

generate, using the hypernetwork, policy weights for a policy;

generate, using the hypernetwork, value function weights for a value function;

calculate a first loss, L_pred, using the mini-batch;

calculate a second loss, L_TD, using the mini-batch;

update the hypernetwork using the first loss and the second loss; and

repeatedly perform the sample through update operations until the hypernetwork has converged.

10 . The apparatus of claim 9 , wherein L_pred is based on predicting a near-optimal action and based on predicting a near-optimal value, and

wherein L_TD is based on moving the predicted near-optimal value toward a current value estimate.

11 . The apparatus of claim 9 , wherein the instructions are further configured to cause the apparatus to at least:

initialize the plurality of system parameter sets to be empty;

use a software model to define a set n-tuples, each n-tuple of the set of n-tuples comprising values of state, action, and reward; and

solve, using an RL solver applied to the set of n-tuples, for the policy and the value function, wherein the policy and the value function form a first set of data, wherein the first set of data comprises an indication of a first context, and the first set of data further comprises a first plurality of system parameters; and

augment the plurality of system parameter sets with the first set of data.

12 . The apparatus of claim 9 , wherein the instructions are further configured to cause the apparatus to at least:

receive a plurality of first trajectories; and

solve for a plurality of first reward parameters and a plurality of first transition dynamics parameters based on the plurality of first trajectories.

13 . The apparatus of claim 12 , wherein the instructions are further configured to cause the apparatus to:

solve for a plurality of first policy parameters based on the plurality of first reward parameters and the plurality of first transition dynamics parameters and based on a plurality of second reward parameters and a plurality of second transition dynamics parameters, and

solve for a plurality of first value parameters based on the plurality of first reward parameters and the plurality of first transition dynamics parameters and based on the plurality of second reward parameters and the plurality of second transition dynamics parameters.

14 . The apparatus of claim 9 , wherein L_pred comprises an expectation of a first mean square loss and a second mean square loss, wherein the first mean square loss is based on a first difference between an experienced value and a ground truth value and the second mean square loss is based on a second difference between a chosen action and a ground truth action.

15 . The apparatus of claim 9 , wherein L_TD comprises an expectation of a third mean square loss, wherein the third mean square loss is based on an obtained reward for moving to a next state summed with a difference of a second experienced value with a ground truth value.

16 . The apparatus of claim 13 , wherein the instructions are further configured to cause the apparatus to at least:

receive a voice command from a user;

map the voice command to a context;

map the context to a hypernetwork input;

obtain the policy by inputting the hypernetwork input to the hypernetwork; and

manipulate, using the policy and in response to the voice command, an object with a robot arm.

17 . A non-transitory computer readable medium storing instructions, the instructions configured to cause an apparatus to at least:

initialize a hypernetwork;

sample a mini-batch of system parameter sets from a plurality of system parameter sets;

generate, using the hypernetwork, policy weights for a policy;

generate, using the hypernetwork, value function weights for a value function;

calculate a first loss, L_pred, using the mini-batch;

calculate a second loss, L_TD, using the mini-batch;

update the hypernetwork using the first loss and the second loss; and

repeatedly perform the sample through update operations until the hypernetwork has converged.

18 . The non-transitory computer readable medium of claim 17 , wherein L_pred is based on predicting a near-optimal action and based on predicting a near-optimal value, and

wherein L_TD is based on moving the predicted near-optimal value toward a current value estimate.

19 . The non-transitory computer readable medium of claim 17 , wherein the instructions are further configured to cause the apparatus to at least:

initialize the plurality of system parameter sets to be empty;

use a software model to define a set n-tuples, each n-tuple of the set of n-tuples comprising values of state, action, and reward; and

solve, using an RL solver applied to the set of n-tuples, for the policy and the value function, wherein the policy and the value function form a first set of data, wherein the first set of data comprises an indication of a first context, and the first set of data further comprises a first plurality of system parameters; and

augment the plurality of system parameter sets with the first set of data.

20 . The non-transitory computer readable medium of claim 17 , wherein the instructions are further configured to cause the apparatus to at least:

receive a voice command from a user;

map the voice command to a context;

map the context to a hypernetwork input;

obtain the policy by inputting the hypernetwork input to the hypernetwork; and

manipulate, using the policy and in response to the voice command, an object with a robot arm.