Systems and methods for safe policy improvement for task oriented dialogues
Embodiments described herein provide safe policy improvement (SPI) in a batch reinforcement learning framework for a task-oriented dialogue. Specifically, a batch reinforcement learning framework for dialogue policy learning is provided, which improves the performance of the dialogue and learns to shape a reward that reasons the invention behind human response rather than just imitating the human demonstration.
1 . A method for causal-aware safety policy improvement in task-oriented learning, comprising:
receiving a training dataset comprising a plurality of dialogues, wherein the plurality of dialogues includes a first dialogue rollout generated according to a latent stochastic behavior policy;
repeatedly sampling the training dataset for a number of times to generate a number of training subsets and a number of validation subsets;
training a task-oriented dialogue model based on a cross-entropy loss using training data in a first training subset of the number of training subsets;
generating, by the task-oriented dialogue model, predicted dialogue rollouts based on dialogue data in a first validation subset of the number of validation subsets;
adding the predicted dialogue rollouts to a pairwise causal learning subset;
sampling a pair of dialogue rollouts from the pairwise causal learning subset; and
training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts.
2 . The method of claim 1 , further comprising:
retrieving a dataset from the number of training subsets or the number of validation subsets; and
training the task-oriented dialogue model by minimizing a cross-entropy of a predicted dialogue action conditioned on a current state of a dialogue according to a target policy using dialogue data from the retrieved dataset.
3 . The method of claim 2 , wherein the predicted dialogue rollouts are repeatedly generated according to the target policy by iterating the number of validation subsets.
4 . The method of claim 1 , wherein the training the task-oriented dialogue model based on a binary cross-entropy loss is performed by repeatedly sampling different pairs of dialogue rollouts from the pairwise causal learning subset and re-training the task-oriented dialogue model based on the binary cross-entropy loss until a convergence is reached in training.
5 . The method of claim 1 , wherein the training the task-oriented dialogue model, based on a binary cross-entropy loss further comprises:
encoding, via three bi-LSTM layers, respectively, a goal, a belief state and a dialogue act or response sequence at each dialogue turn of each of the sampled pair of dialogue rollouts into three encoded representations;
concatenating the three encoded representations;
feeding the concatenated encoded representations to one or more feed-forward layers that generates a reward prediction for each dialogue turn;
summing generated reward predictions into a dialogue reward for each one of the sampled pair of dialogue rollouts;
computing the preferred probability between the pair of dialogue rollouts based on dialogue rewards corresponding to the sampled pair of dialogue rollouts; and
computing the binary cross-entropy loss between the preferred probability between the pair of dialogue rollouts and the normalized metric score based on the pair of dialogue rollouts.
6 . The method of claim 5 , wherein the preferred probability between the pair of dialogue rollouts is computed using normalization or a softmax function.
7 . The method of claim 1 , further comprising:
repeatedly training the task-oriented dialogue model based on a policy optimization loss that optimizes over the target policy using the training dataset until a training convergence is reached.
8 . The method of claim 7 , wherein the policy optimization loss is computed by:
generating, by the task-oriented dialogue model, a first predicted action distribution based on a current state of a dialogue according to a target policy;
computing a first discounted sum of future reward based on a discount parameter and a reward function of actions and states of the dialogue according to the latent behavior policy;
computing a first loss objective based on a first expectation of the first discounted sum of future reward and the first predicted action distribution, wherein the first expectation is taken over a probability distribution of the states and the actions according to the latent stochastic behavior policy;
generating, by the task-oriented dialogue model, a second predicted action distribution based on a current observation from a time series of observations according to the target policy;
computing a second discounted sum of future reward based on the discount parameter and the reward function for a specific rollout, wherein the second discounted sum of future reward is a collapsed near-deterministic approximation of the first discounted sum of future reward;
computing a second loss objective based on a second expectation of the second discounted sum of future reward and the second predicted action distribution, wherein the second expectation is taken over an average of the observations across the training dataset; and
computing a sum of the first loss objective and the second loss objective.
9 . The method of claim 8 , further comprising:
computing a gradient update component based on a learnt reward from the reward function of actions and states of the dialogue and a gradient of the target policy of the actions conditioned on the states and parameters of the task-oriented dialogue model; and
updating the parameters of the task-oriented dialogue model using the gradient update component.
10 . The method of claim 1 , further comprising:
randomly selecting a dialogue turn during validation of the trained task-oriented dialogue model; and
receiving a set of manually created evaluation scores of a prediction on the dialogue turn from a plurality of evaluators.
11 . A system for causal-aware safety policy improvement in task-oriented learning, the system comprising:
a communication interface receiving a training dataset comprising a plurality of dialogues, wherein the plurality of dialogues includes a first dialogue rollout generated according to a latent stochastic behavior policy;
a memory storing a plurality of processor-executable instructions; and
a processor reading the plurality of processor-executable instructions from the memory to perform operations comprising:
repeatedly sampling the training dataset for a number of times to generate a number of training subsets and a number of validation subsets;
training a task-oriented dialogue model based on an entropy loss using training data in a first training subset of the number of training subsets;
generating, by the task-oriented dialogue model, predicted dialogue rollouts from dialogue data in a first validation subset of the number of validation subsets;
adding the predicted dialogue rollouts to a pairwise causal learning subset;
sampling a pair of dialogue rollouts from the pairwise causal learning subset; and
training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts.
12 . The system of claim 11 , wherein the operations further comprise:
retrieving a dataset from the number of training subsets or the number of validation subsets; and
training the task-oriented dialogue model by minimizing an entropy of a predicted dialogue action conditioned on a current state of a dialogue according to a target policy using dialogue data from the retrieved dataset.
13 . The system of claim 12 , wherein the predicted dialogue rollouts are repeatedly generated according to the target policy by iterating the number of validation subsets.
14 . The system of claim 11 , wherein the operation of training the task-oriented dialogue model based on a binary cross-entropy loss is performed by repeatedly sampling different pairs of dialogue rollouts from the pairwise causal learning subset and re-training the task-oriented dialogue model based on the binary cross-entropy loss until a convergence is reached in training.
15 . The system of claim 11 , wherein the operation of training the task-oriented dialogue model based on a binary cross-entropy loss further comprises:
encoding, via three bi-LSTM layers, respectively, a goal, a belief state and a dialogue act or response sequence at each dialogue turn of each of the sampled pair of dialogue rollouts into three encoded representations;
concatenating the three encoded representations;
feeding the concatenated encoded representations to one or more feed-forward layers that generates a reward prediction for each dialogue turn;
summing generated reward predictions into a dialogue reward for each one of the sampled pair of dialogue rollouts;
computing the preferred probability between the pair of dialogue rollouts based on dialogue rewards corresponding to the sampled pair of dialogue rollouts; and
computing the binary cross-entropy loss between the preferred probability between the pair of dialogue rollouts and the normalized metric score based on the pair of dialogue rollouts.
16 . The system of claim 15 , wherein the preferred probability between the pair of dialogue rollouts is computed using normalization or a softmax function.
17 . The system of claim 11 , wherein the operations further comprise:
generating, by the task-oriented dialogue model, a first predicted action distribution based on a current state of a dialogue according to a target policy;
computing a first discounted sum of future reward based on a discount parameter and a reward function of actions and states of the dialogue according to the latent behavior policy;
computing a first loss objective based on a first expectation of the first discounted sum of future reward and the first predicted action distribution, wherein the first expectation is taken over a probability distribution of the states and the actions according to the latent stochastic behavior policy;
generating, by the task-oriented dialogue model, a second predicted action distribution based on a current observation from a time series of observations according to the target policy;
computing a second discounted sum of future reward based on the discount parameter and the reward function for a specific rollout, wherein the second discounted sum of future reward is a collapsed near-deterministic approximation of the first discounted sum of future reward;
computing a second loss objective based on a second expectation of the second discounted sum of future reward and the second predicted action distribution, wherein the second expectation is taken over an average of the observations across the training dataset; and
computing a sum of the first loss objective and the second loss objective.
18 . The system of claim 17 , wherein the operations further comprise:
computing a gradient update component based on a learnt reward from the reward function of actions and states of the dialogue and a gradient of the target policy of the actions conditioned on the states and parameters of the task-oriented dialogue model; and
updating the parameters of the task-oriented dialogue model using the gradient update component.
19 . The system of claim 11 , wherein the operations further comprise:
randomly selecting a dialogue turn during validation of the trained task-oriented dialogue model; and
receiving a set of manually created evaluation scores of a prediction on the dialogue turn from a plurality of evaluators.
20 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for causal-aware safety policy improvement in task-oriented learning, the instructions being executed by a processor to perform operations comprising:
receiving a training dataset comprising a plurality of dialogues, wherein the plurality of dialogues includes a first dialogue rollout generated according to a latent stochastic behavior policy;
repeatedly sampling the training dataset for a number of times to generate a number of training subsets and a number of validation subsets;
training a task-oriented dialogue model based on an entropy loss using training data in a first training subset of the number of training subsets;
generating, by the task-oriented dialogue model, predicted dialogue rollouts from dialogue data in a first validation subset of the number of validation subsets;
adding the predicted dialogue rollouts to a pairwise causal learning subset;
sampling a pair of dialogue rollouts from the pairwise causal learning subset; and
training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts.