EXPERIENCE SELECTION IN REINFORCEMENT LEARNING
Techniques described herein include selecting experience data for use when training or retraining a model. In one example, this disclosure describes a method that includes generating a plurality of trajectories, each comprising a contiguous sequence of instances of experience data, where each instance of experience data in the contiguous sequence has an error value associated that instance of experience data; determining, for each of the trajectories, a sorted order of the instances of experience data, wherein the sorted order is based on the error value associated with each of the instances of experience data; selecting, based on a distribution function applied to the sorted order of the instances of experience data in at least one of the trajectories, a subset of instances of the experience data; and retraining a reinforcement learning model, using the subset of instances of experience data, to predict an optimal action to take in a state.
1 . A computing system comprising a storage device and processing circuitry having access to the storage device, wherein the processing circuitry is configured to:
generate a plurality of trajectories, each comprising a contiguous sequence of instances of experience data, where each instance of experience data in the contiguous sequence has an error value associated with that instance of experience data;
determine, for each of the trajectories, a sorted order of the instances of experience data, wherein the sorted order is based on the error value associated with each of the instances of experience data;
select, based on a distribution function applied to the sorted order of the instances of experience data in at least one of the trajectories, a subset of instances of the experience data; and
retrain a reinforcement learning model, using the subset of instances of experience data, to predict an optimal action to take in a state.
2 . The computing system of claim 1 , wherein the plurality of trajectories includes a first trajectory and a second trajectory, and wherein to select the subset of instances, the processing circuitry is further configured to:
select a first subset of instances of the experience data from the first trajectory by applying a first distribution function to the sorted order of the instances of experience data in that first trajectory; and
select a second subset of instances of the experience data from the second trajectory by applying a second distribution function to the sorted order of the instances of experience data in that second trajectory.
3 . The computing system of claim 2 ,
wherein the first distribution function is the same as the second distribution function.
4 . The computing system of claim 1 , wherein each of the plurality of trajectories correspond to a different one of a plurality of episodes, each episode ending with an instance of experience data having a termination condition, and wherein to generate the plurality of trajectories, the processing circuitry is further configured to:
generate the plurality of episodes.
5 . The computing system of claim 4 , wherein to determine the sorted order, the processing circuitry is further configured to:
determine, for each of the episodes, the sorted order of the instances of experience data.
6 . The computing system of claim 1 , wherein to select the subset of instances, the processing circuitry is further configured to:
select, at a first frequency rate, instances of the experience data having high error; and
select, at a second frequency rate, instances of the experience data having less than high error, wherein the first frequency rate is greater than the second frequency rate.
7 . The computing system of claim 6 , wherein to select the subset of instances, the processing circuitry is further configured to:
apply at least one of a power law distribution function or a truncated half gaussian distribution to the sorted order of the instances of experience data.
8 . The computing system of claim 1 ,
wherein the error value associated with each of the instances of experience data is a temporal difference error.
9 . The computing system of claim 1 ,
wherein the error value associated with each of the instances of experience data is a difference between an observed reward received during a simulation and an expected reward based on a prediction made by the reinforcement learning model.
10 . The computing system of claim 1 , wherein the processing circuitry is further configured to:
send control signals to a production system once the reinforcement learning model is retrained, the control signals instructing the production system to take actions specified in the control signals and wherein the control signals are based on outputs generated by the retrained reinforcement learning model.
11 . A method comprising:
generating, by a computing system, a plurality of trajectories, each comprising a contiguous sequence of instances of experience data, where each instance of experience data in the contiguous sequence has an error value associated with that instance of experience data;
determining, by the computing system and for each of the trajectories, a sorted order of the instances of experience data, wherein the sorted order is based on the error value associated with each of the instances of experience data;
selecting, by the computing system and based on a distribution function applied to the sorted order of the instances of experience data in at least one of the trajectories, a subset of instances of the experience data; and
retraining, by the computing system, a reinforcement learning model, using the subset of instances of experience data, to predict an optimal action to take in a state.
12 . The method of claim 11 , wherein the plurality of trajectories includes a first trajectory and a second trajectory, and wherein selecting the subset of instances includes:
selecting a first subset of instances of the experience data from the first trajectory by applying a first distribution function to the sorted order of the instances of experience data in that first trajectory; and
selecting a second subset of instances of the experience data from the second trajectory by applying a second distribution function to the sorted order of the instances of experience data in that second trajectory.
13 . The method of claim 12 ,
wherein the first distribution function is the same as the second distribution function.
14 . The method of claim 11 , wherein each of the plurality of trajectories correspond to a different one of a plurality of episodes, each episode ending with an instance of experience data having a termination condition, and wherein generating the plurality of trajectories includes:
generating the plurality of episodes.
15 . The method of claim 14 , wherein determining the sorted order includes:
determining, for each of the episodes, the sorted order of the instances of experience data.
16 . The method of claim 11 , wherein selecting the subset of instances includes:
selecting, at a first frequency rate, instances of the experience data having high error; and
selecting, at a second frequency rate, instances of the experience data having less than high error, wherein the first frequency rate is greater than the second frequency rate.
17 . The method of claim 16 , wherein selecting the subset of instances includes:
applying a power law distribution function to the sorted order of the instances of experience data.
18 . The method of claim 16 , wherein selecting the subset of instances includes:
applying a truncated half gaussian distribution to the sorted order of the instances of experience data.
19 . The method of claim 11 ,
wherein the error value associated with each of the instances of experience data is a temporal difference error.
20 . A non-transitory computer-readable medium comprising instructions that, when executed, cause processing circuitry of a computing system to:
generate a plurality of trajectories, each comprising a contiguous sequence of instances of experience data, where each instance of experience data in the contiguous sequence has an error value associated with that instance of experience data;
determine, for each of the trajectories, a sorted order of the instances of experience data, wherein the sorted order is based on the error value associated with each of the instances of experience data;
select, based on a distribution function applied to the sorted order of the instances of experience data in at least one of the trajectories, a subset of instances of the experience data; and
retrain a reinforcement learning model, using the subset of instances of experience data, to predict an optimal action to take in a state.