Method and system for fine-tuning neural conditional language models using constraints
A processor-implemented method for fine-tuning a pre-trained neural conditional language model to perform a downstream task. A pre-trained conditional language model and at least one target constraint for satisfying a task-related control objective are received. A neural model is trained to approximate a target conditional model that optimally reconciles a distance from the pre-trained conditional language model and the control objective across multiple contexts.
1 . A method implemented by a processor for fine-tuning a pre-trained neural conditional language model to perform a downstream task, the method comprising:
receiving the pre-trained conditional language model having attributes with existing probability distributions conditioned on contexts;
receiving at least one target constraint for satisfying a task-related control objective associated with the downstream task, the target constraint specifying an expectation of a target attribute;
using the processor, training a neural model to approximate a target conditional model that reconciles a distance from the pre-trained conditional language model and the control objective across multiple contexts; and
outputting the trained neural model representing the pre-trained neural conditional language model fine-tuned to perform the downstream task;
wherein said training the neural model comprises:
for each of one or more iterations:
sampling contexts from a distribution of contexts;
computing an unconditional energy-based model (EBM) for each sampled context using the pre-trained conditional language model and the target constraint; and
updating the neural model over the sampled contexts using the computed unconditional EBMs.
2 . The method of claim 1 , wherein the trained neural model comprises one of a sequence-to-sequence (seq2seq) model, an encoder and a decoder.
3 . The method of claim 1 , wherein the trained neural model is trained to approximate a target conditional model that optimally reconciles a distance from the pre-trained conditional language model and the control objective on average across multiple contexts.
4 . The method of claim 1 , wherein the trained neural model comprises a sequence-to-sequence (seq2seq) model, and wherein said training the seq2seq model comprises:
initializing a reference policy; and
training the reference policy by stochastic gradient descent using a loss gradient that minimizes a distance from the pre-trained conditional language model and the control objective across multiple contexts to provide the trained neural seq2seq model.
5 . The method of claim 4 , wherein the loss gradient minimizes a distance from the pre-trained conditional language model and the control objective on average across multiple contexts to provide the trained neural seq2seq model.
6 . The method of claim 4 , wherein the loss gradient minimizes an expected cross-entropy CE between the reference policy and multiple target distributions p c 's, where each target distribution p c is a normalization of an unconditional energy-based model (EBM) P c mapped by the target conditional model to a context c, and where the expected cross-entropy is over the distribution of contexts, the distribution of contexts being a distribution τ(c) of contexts c over a set of contexts C.
7 . The method of claim 6 , wherein the trained neural model comprises a sequence-to-sequence (seq2seq) model
wherein said sampling contexts comprises sampling N contexts from the distribution τ(c) of contexts c;
wherein said computing an unconditional energy-based model comprises computing the unconditional energy-based model (EBM) P c for each sampled context c over the distribution τ(c) of contexts c using the pre-trained conditional language model and the target constraint; and
wherein said updating the neural model comprises updating the neural model over the N contexts by importance sampling using, for each context, M samples from the reference policy.
8 . The method of claim 7 , wherein the distribution τ(c) of contexts c is provided from a set of source documents.
9 . The method of claim 7 , wherein the N sample contexts are source documents from the set of contexts C.
10 . The method of claim 9 , wherein the source documents are provided from one or more of a dataset, an encoder encoding an input sequence, or a portion of the pretrained model.
11 . The method of claim 6 , wherein the unconditional EBM P c corresponds to the target distribution p c ; and
wherein the target distribution p c is conditioned on the context c based on probabilities provided by the pre-trained conditional language model for the context a(x/c) and the target constraint for the context b(x/c).
12 . The method of claim 11 , wherein the target constraint is specified by a constraint satisfaction score that is combined with the probabilities provided by the pre-trained conditional language model for the context.
13 . The method of claim 12 , wherein the target constraint comprises one or more of a pointwise constraint, a binary constraint and a distributional constraint.
14 . The method of claim 4 ,
wherein said updating the neural model comprises:
for each of N sampled contexts, sampling M samples x from the reference policy based on the context;
estimating a normalization for the unconditional energy-based model (EBM) P c for each context c over the M samples using the computed unconditional EBM for the context; and
updating parameters of the reference policy by applying the estimated normalization for each context-sample pair to the loss gradient;
where M and N are hyperparameters.
15 . The method of claim 14 , wherein:
the estimated normalization for each context-sample pair is stored in a buffer during each iteration; and
said applying the estimated normalization for each context-sample pair to the loss gradient comprises:
shuffling the buffer; and
iterating over the shuffled buffer.
16 . The method of claim 15 , wherein said estimating a normalization comprises: computing a score for each sample x using the computed unconditional EBM for the context c; and
computing the estimated normalization using the computed scores.
17 . The method of claim 16 , wherein the estimated normalization comprises a normalizing constant or partition function.
18 . The method of claim 14 , wherein said estimating a normalization further comprises:
computing the estimated normalization over the M obtained samples by importance sampling; and/or
reweighting the obtained samples by their likelihood according to the reference policy.
19 . The method of claim 1 , wherein the reference policy is initialized using the pre-trained conditional language model.
20 . The method of claim 1 , further comprising:
determining if the reference policy has converged with the target conditional model; and
ending the training if the reference policy has converged with the target conditional model.
21 . The method of claim 1 , wherein the downstream task is one of a summarization task, a code generation task, a dialogue task, and a translation task.
22 . The method of claim 1 , wherein the downstream task is a summarization task and the context is provided by processing a document to be summarized, and the constraint is based on factual correctness of the summarized document.
23 . The method of claim 1 , wherein the downstream task is a translation task and the context is provided by processing a document to be translated, and the constraint is based on consistency of terminology.
24 . A method for generating text comprising:
receiving a context by a trained neural language model trained according to the method of claim 1 ;
the trained neural language model generating text in response to the received context.
25 . A method for generating text comprising:
receiving an input sequence;
processing the input sequence to determine a context; and
processing the context by a trained neural model trained according to the method of claim 1 to generate text in response to the context.
26 . A non-transitory computer-readable medium having executable instructions stored thereon for causing a processor and a memory to implement a method for fine-tuning a pre-trained neural conditional language model to perform a downstream task, the method comprising:
receiving the pre-trained conditional language model having attributes with existing probability distributions conditioned on contexts;
receiving at least one target constraint for satisfying a task-related control objective associated with the downstream task, the target constraint specifying an expectation of a target attribute;
training a neural model using the processor to approximate a target conditional model that reconciles a distance from the pre-trained conditional language model and the control objective across multiple contexts; and
outputting the trained neural model;
wherein said training the neural model comprises:
for each of one or more iterations:
sampling contexts from a distribution of contexts;
computing an unconditional energy-based model (EBM) for each sampled context using the pre-trained conditional language model and the target constraint; and
updating the neural model over the sampled contexts using the computed unconditional EBMs.
27 . A processor-based system for fine-tuning a pre-trained neural conditional language model to perform a downstream task, comprising:
an energy-based model (EBM) computing block configured to compute using one or more processors an energy-based model based on a received pre-trained conditional language model having attributes with existing probability distributions conditioned on contexts and at least one received target constraint for satisfying a task-related control objective, the target constraint specifying an expectation of a target attribute;
a context sampling block configured to sample using one or more processors a plurality of contexts;
a model output sampling block configured to sample using one or more processors a plurality of model outputs associated with each sampled context;
a normalization estimating block configured to estimate using one or more processors a normalization based on said computed energy-based model and said sampled plurality of model outputs; and
a model updating block for training a neural model using one or more processors to approximate using one or more processors a target conditional model that reconciles a distance from the pre-trained conditional language model and the control objective across multiple contexts based on a loss gradient determined using said estimated normalization; the model updating block being further configured to output the trained neural model.
28 . The system of claim 27 , wherein said model updating block is configured to iteratively update parameters of the neural model.
29 . The system of claim 28 , further comprising:
a memory for storing said sampled plurality of contexts, said sample plurality of model outputs, and said estimated normalizations;
wherein the stored estimated normalizations are associated with the sampled plurality of contexts and the sample plurality of model outputs in the memory.
30 . The system of claim 29 , wherein said model updating block is configured to iteratively update parameters of the neural model updating block by:
shuffling at least the stored estimated normalizations; and
computing the loss gradient using the shuffled estimated normalizations.
31 . The system of claim 30 , wherein the trained neural model comprises at least a portion of a sequence-to-sequence (seq2seq) model.
32 . The system of claim 31 , wherein the trained neural model is trained to approximate a target conditional model that optimally reconciles a distance from the pre-trained conditional language model and the control objective on average across multiple contexts.