Reinforcement learning with information retrieval feedback
In one example aspect, the present disclosure provides an example computer-implemented method for generating feedback signals for training a machine-learned agent model. The example method can include obtaining an output of a machine-learned agent model, the output including a next state feature generated by the machine-learned agent model based on a sequence of preceding states. The example method can include processing, using a machine-learned reward model, the output and the sequence of preceding states to generate a quality indicator indicating a quality of the next state feature in view of the preceding states. The machine-learned reward model could be trained by retrieving reference data from a reference data source and computing one or more quality indicators in view of a respective training input and output(s), and the reference data. The example method can include outputting the quality indicator to a model trainer for updating the machine-learned agent model.
1 . A computer-implemented method for generating feedback signals for training a machine-learned agent model, the method comprising:
obtaining, by a computing system comprising one or more processors, an output of a machine-learned agent model, wherein the output comprises a next state feature generated by the machine-learned agent model based on a sequence of preceding states;
processing, by the computing system and using a machine-learned reward model, the output and the sequence of preceding states to generate a quality indicator that indicates a quality of the next state feature in view of the preceding states,
wherein the machine-learned reward model was trained by:
obtaining a sampled plurality of outputs;
processing, using the machine-learned reward model, each of the sampled plurality of outputs to generate a plurality of quality indicators;
processing, using a machine-learned supervisor model, each of the sampled plurality of outputs in view of reference data retrieved from a reference data source to indicate a preferred sampled output; and
training the machine-learned reward model to generate a more preferential quality signal for the preferred sampled output; and
outputting, by the computing system, the quality indicator to a model trainer for updating the machine-learned agent model based on the quality indicator.
2 . The computer-implemented method of claim 1 , wherein the quality indicator comprises a binary indicator.
3 . The computer-implemented method of claim 1 , wherein:
the next state feature indicates an action to perform using a tool external to the machine-learned agent model;
a next state comprises the next state feature and a result obtained by performing the action using the tool; and
the next state is added to the sequence of preceding states for future iterations.
4 . The computer-implemented method of claim 1 , comprising:
retrieving, by the computing system and from a reference data source, reference data associated with the preceding states; and
processing, by the computing system and using the machine-learned reward model, the output and the sequence of preceding states in view of the reference data to generate the quality indicator.
5 . The computer-implemented method of claim 1 , comprising:
retrieving, by the computing system and from a reference data source, reference data associated with the preceding states;
processing, by the computing system and using the machine-learned supervisor model, the output, the sequence of preceding states, and the reference data to generate a reference quality indicator that indicates the preferred sampled output; and
training, by the computing system, the machine-learned reward model based on the reference quality indicator.
6 . The computer-implemented method of claim 5 , wherein the reference data is obtained by:
retrieving, by the computing system, content from a plurality of web resources;
summarizing, by the computing system, the content; and
composing, by the computing system, the reference data by concatenating the summarized content.
7 . The computer-implemented method of claim 5 , comprising:
obtaining, by the computing system, the sampled plurality of outputs from the machine-learned agent model.
8 . The computer-implemented method of claim 7 , wherein:
the plurality of quality indicators form a quality indicator logit;
the preferred sampled output is indicated by a one-hot vector across the sampled of plurality of outputs; and
the method comprises:
determining, by the computing system, a cross-entropy loss between the quality indicator logit and the one-hot vector.
9 . The computer-implemented method of claim 1 , comprising: training, by the computing system and using the model trainer, the machine-learned agent model based on the quality indicator.
10 . The computer-implemented method of claim 9 , wherein training the machine-learned agent model comprises:
updating, by the computing system, one or more parameters of the machine-learned agent model to increase an expected reward.
11 . The computer-implemented method of claim 9 , wherein training the machine-learned agent model comprises:
updating, by the computing system, one or more parameters of the machine-learned agent model to decrease an expected length of a sequence of states.
12 . The computer-implemented method of claim 1 , wherein the machine-learned agent model comprises a machine-learned sequence-to-sequence model configured for natural language processing.
13 . The computer-implemented method of claim 12 , wherein the machine-learned reward model comprises a machine-learned sequence-to-sequence model configured for natural language processing.
14 . The computer-implemented method of claim 5 , wherein the machine-learned supervisor model comprises a machine-learned sequence-to-sequence model configured for natural language processing.
15 . A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining an output of a machine-learned agent model, wherein the output comprises a next state feature generated by the machine-learned agent model based on a sequence of preceding states;
processing, using a machine-learned reward model, the output and the sequence of preceding states to generate a quality indicator that indicates a quality of the next state feature in view of the preceding states,
wherein the machine-learned reward model was trained by:
obtaining a sampled plurality of outputs;
processing, using the machine-learned reward model, each of the sampled plurality of outputs to generate a plurality of quality indicators;
processing, using a machine-learned supervisor model, each of the sampled plurality of outputs in view of reference data retrieved from a reference data source to indicate a preferred sampled output; and
training the machine-learned reward model to generate a more preferential quality signal for the preferred sampled output; and
outputting the quality indicator to a model trainer for updating the machine-learned agent model based on the quality indicator.
16 . The computing system of claim 15 , the operations comprising:
retrieving, from a reference data source, reference data associated with the preceding states;
processing, using the machine-learned supervisor model, the output, the sequence of preceding states, and the reference data to generate a reference quality indicator that indicates the preferred sampled output; and
training the machine-learned reward model based on the reference quality indicator.
17 . The computing system of claim 15 , wherein the reference data is obtained by:
retrieving content from a plurality of web resources;
summarizing the content; and
composing the reference data by concatenating the summarized content.
18 . The computing system of claim 16 , the operations comprising:
obtaining the sampled plurality of outputs from the machine-learned agent model.
19 . The computing system of claim 18 , wherein:
the plurality of quality indicators form a quality indicator logit;
the preferred sampled output is indicated by a one-hot vector across the sampled of plurality of outputs; and
the operations comprise:
determining a cross-entropy loss between the quality indicator logit and the one-hot vector.
20 . One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to cause a computing system to perform operations, the operations comprising:
obtaining an output of a machine-learned agent model, wherein the output comprises a next state feature generated by the machine-learned agent model based on a sequence of preceding states;
processing, using a machine-learned reward model, the output and the sequence of preceding states to generate a quality indicator that indicates a quality of the next state feature in view of the preceding states,
wherein the machine-learned reward model was trained by:
obtaining a sampled plurality of outputs;
processing, using the machine-learned reward model, each of the sampled plurality of outputs to generate a plurality of quality indicators;
processing, using a machine-learned supervisor model, each of the sampled plurality of outputs in view of reference data retrieved from a reference data source to indicate a preferred sampled output; and
training the machine-learned reward model to generate a more preferential quality signal for the preferred sampled output; and
outputting the quality indicator to a model trainer for updating the machine-learned agent model based on the quality indicator.