Reinforced generation: reinforcement learning for text and knowledge graph bi-directional generation using pretrained language models
Obtain access to a pretrained encoder-decoder language model. Using a dataset including a plurality of text-graph pairs, carry out first fine-tuning training on the pre-trained language model by minimizing cross-entropy loss. A text portion of each text-graph pair includes a list of text tokens and a graph portion of each text-graph pair includes a list of graph tokens. The first fine-tuning training results in an intermediate model. Carry out second fine-tuning training on the intermediate model, by reinforcement learning, to obtain a final model. Make the final model available for deployment.
1 . A method comprising:
obtaining access to a pretrained encoder-decoder language model;
obtaining a list of graph tokens by linearizing a plurality of sets of triples, each set of triples comprising a subject, a relationship, and an object, into a corresponding entry in the list of graph tokens with marker tokens for the subject, the relationship, and the object;
using a dataset comprising a plurality of text-graph pairs, carrying out first fine-tuning training on the pre-trained language model by minimizing cross-entropy loss, wherein a text portion of each text-graph pair comprises a list of text tokens and a graph portion of each text-graph pair comprises the list of graph tokens, the first fine-tuning training resulting in an intermediate model;
carrying out second fine-tuning training on the intermediate model, by reinforcement learning and applying self-critical sequence training, to obtain a final model, wherein the first and second fine tuning training each comprise fine tuning training to implement both knowledge graph to text generation and text to knowledge graph generation; and
making the final model available for deployment.
2 . The method of claim 1 , where applying self-critical sequence training comprises minimizing a loss function L SCST in accordance with:
∇
θ
ℒ
SCST
∝
-
(
R
(
x
^
T
)
-
R
(
x
T
*
)
)
∇
θ
log
p
θ
(
x
^
T
)
wherein:
θ is a model parameterizing factor;
p is a policy represented by the intermediate model parameterized by θ;
R is a reward;
{circumflex over (x)} T is a generated sequence sampled from the intermediate model; and
x
T
*
is generated sequence obtained by greedy max.
3 . The method of claim 1 , wherein the minimizing of the cross-entropy loss is carried out in accordance with:
ℒ
CE
G
=
[
-
log
p
θ
T
2
G
(
x
G
)
]
wherein:
ℒ
CE
G
is the cross-entropy loss;
T2G denotes graph-to-text;
θ is a model parameterizing factor;
p is a policy represented by the intermediate model parameterized by θ;
x G is text to be generated;
is the dataset;
and is expectation.
4 . The method of claim 1 , further comprising:
deploying the final model; and
generating a corresponding knowledge graph from obtained input text in accordance with the deployed final model.
5 . The method of claim 4 , further comprising controlling a physical system in accordance with the generated knowledge graph.
6 . The method of claim 1 , wherein the minimizing of cross-entropy loss is carried out in accordance with:
ℒ
CE
T
=
[
-
log
p
θ
G
2
T
(
x
T
)
]
wherein:
ℒ
CE
T
is the cross-entropy loss;
G2T denotes graph-to-text;
θ is a model parameterizing factor;
p is a policy represented by the intermediate model parameterized by θ;
x T is text to be generated;
is the dataset;
and is expectation.
7 . The method of claim 1 , further comprising:
deploying the final model; and
generating corresponding text from an obtained input knowledge graph in accordance with the deployed final model.
8 . The method of claim 1 , further comprising:
deploying the final model;
generating corresponding text from an obtained input knowledge graph in accordance with the deployed final model; and
generating a corresponding knowledge graph from obtained input text in accordance with the deployed final model.
9 . The method of claim 8 , further comprising controlling a physical system in accordance with the generated knowledge graph.
10 . A non-transitory computer readable medium comprising computer executable instructions which when executed by a computer cause the computer to perform the method of:
obtaining access to a pretrained encoder-decoder language model;
obtaining a list of graph tokens by linearizing a plurality of sets of triples, each set of triples comprising a subject, a relationship, and an object, into a corresponding entry in the list of graph tokens with marker tokens for the subject, the relationship, and the object;
using a dataset comprising a plurality of text-graph pairs, carrying out first fine-tuning training on the pre-trained language model by minimizing cross-entropy loss, wherein a text portion of each text-graph pair comprises a list of text tokens and a graph portion of each text-graph pair comprises the list of graph tokens, the first fine-tuning training resulting in an intermediate model;
carrying out second fine-tuning training on the intermediate model, by reinforcement learning and applying self-critical sequence training, to obtain a final model, wherein the first and second fine tuning training each comprise fine tuning training to implement both knowledge graph to text generation and text to knowledge graph generation; and
making the final model available for deployment.
11 . An apparatus comprising:
a memory; and
at least one processor, coupled to the memory, and operative to:
obtain access to a pretrained encoder-decoder language model;
obtain a list of graph tokens by linearizing a plurality of sets of triples, each set of triples comprising a subject, a relationship, and an object, into a corresponding entry in the list of graph tokens with marker tokens for the subject, the relationship, and the object;
using a dataset comprising a plurality of text-graph pairs, carry out first fine-tuning training on the pre-trained language model by minimizing cross-entropy loss, wherein a text portion of each text-graph pair comprises a list of text tokens and a graph portion of each text-graph pair comprises the list of graph tokens, the first fine-tuning training resulting in an intermediate model;
carry out second fine-tuning training on the intermediate model, by reinforcement learning and applying self-critical sequence training, to obtain a final model, wherein the first and second fine tuning training each comprise fine tuning training to implement both knowledge graph to text generation and text to knowledge graph generation; and
make the final model available for deployment.