Length-controlled text generation using a text processing model
The disclosure herein describes training a text processing model to generate model output text data using input text data and a sentence count. A training data entry including input text data and output text data is obtained. A sentence count of the output text data is determined, and the output text data is labeled with a sentence count label and a sentence number label. Model output text data is generated with a text processing model using the input text data and determined sentence count as input data. Loss data associated with a difference between the generated model output text data and the labeled output text data is used to adjust the text processing model, which enables the model to be trained to produce output text data with a target sentence count in a computationally efficient manner.
1 . A system comprising:
a processor; and
a memory comprising computer program code for execution by the processor, wherein the computer program code, when executed by the processor, causes the system to:
obtain input text data and associated output text data;
determine a sentence count of the output text data, wherein the sentence count indicates a number of sentences within the output text data;
label the output text data with a sentence count label and a sentence number label using the determined sentence count; and
iteratively train a text processing machine learning (ML) model over multiple training iterations based on training data that includes the input text data, the output text data, and the sentence count, wherein iteratively training the text processing ML model includes, during each of the multiple training iterations, training the text processing ML model to generate model output text data from the input text data based on the sentence count, determining loss data based on a difference between the model output text data and the output text data, and adjusting weight values of the text processing ML model using the determined loss data.
2 . The system of claim 1 , wherein the computer program code, when executed by the processor, further causes the system to:
iteratively train a sentence count prediction ML model over the multiple training iterations based on the training data, wherein iteratively training the sentence count prediction ML model includes, during each of the multiple training iterations, training the sentence count prediction ML model to predict a sentence count of the model output text data generated by the text processing ML model, determining sentence count prediction loss data based on a difference between the predicted sentence count and an actual sentence count of the model output text data generated by the text processing ML model, and adjusting weight values of the sentence count prediction ML model using the determined sentence count prediction loss data.
3 . The system of claim 2 , wherein the text processing ML model and the sentence count prediction ML model share an encoder layer, and
wherein, during each of the multiple training iterations, the text processing ML model uses the encoder layer to generate model output text data and the sentence count prediction ML model uses the encoder layer to predict the sentence count of the model output text data generated by the text processing ML model.
4 . The system of claim 2 , wherein the weight values of the text processing ML model and the weight values of the sentence count prediction ML model are adjusted in parallel.
5 . The system of claim 1 , wherein labeling the output text data includes:
inserting a special character and the determined sentence count at a beginning of the output text data as the sentence count label;
determining a sentence number value of a sentence in the output text data using a position of the sentence in the output text data relative to the beginning of the output text data; and
inserting the special character and the determined sentence number value at a beginning of the sentence as sentence number label of the sentence number labels.
6 . The system of claim 1 , wherein the output text data both translates and summarizes the input text data such that the output text data is in a different language and of a shorter length than the input text data.
7 . One or more computer storage media having computer-executable instructions that, upon execution by a processor of a system, cause the system to at least:
obtain input text data and associated output text data;
determine a token count of the output text data, wherein the token count indicates a number of tokens in the output text data;
determine reverse position values of the tokens in the output text data, wherein the reverse position values of the tokens in the output text data indicate positions of the tokens in the output text data in a reverse order;
iteratively train a text processing machine learning (ML) model over multiple training iterations based on training data that includes the input text data, the output text data, the token count, and the reverse position values, wherein iteratively training the text processing ML model includes, during each of the multiple training iterations, training the text processing ML model to generate model output text data from the input text data based on the reverse position values, determining loss data based on a difference between the model output text data and the output text data, and adjusting weight values of the text processing ML model using the determined loss data.
8 . The one or more computer storage media of claim 7 , wherein the computer-executable instructions, upon execution by the processor, further cause the system to:
iteratively train a token count prediction ML model over the multiple training iterations based on the training data, wherein iteratively training the token count prediction ML model includes, during each of the multiple training iterations, training the token count prediction ML model to predict a token count of the model output text data generated by the text processing ML model, determining token count prediction model loss data based on a difference between the predicted token count and an actual token count of the model output text data generated by the text processing ML model, and adjusting weight values of the token count prediction ML model using the determined token count prediction model loss data.
9 . The one or more computer storage media of claim 8 , wherein the text processing ML model and the token count prediction ML model share an encoder layer, and
wherein, during each of the multiple training iterations, the text processing ML model uses the encoder layer to generate the model output text data and the token count prediction ML model uses the encoder layer to predict token count of the model output text data generated by the text processing ML model.
10 . The one or more computer storage media of claim 8 , wherein the weight values of text processing model and the weight values of the token count prediction model are adjusted in parallel.
11 . The one or more computer storage media of claim 7 , wherein the output text data both translates and summarizes the input text data such that the output text data is in a different language and of a shorter length than the input text data.
12 . The one or more computer storage media of claim 7 , wherein training the text processing ML model to generate the model output text data during each of the multiple training iterations includes training an input embedding layer of the text processing ML model to generate an input embedding based on the input text data, training a position embedding layer of the text processing ML model to generate an output position embedding based on the token count and the reverse position values, and training encoding/decoding layers of the text processing ML model to generate the model output text based on the input embedding and the output position embedding.
13 . The one or more computer storage media of claim 7 , wherein adjusting the weight values of the text processing ML model using the determined loss data comprises adjusting weight values of a position embedding layer of the text processing ML model using the loss data, the reverse position values being an input to the position embedding layer when training the text processing ML model to generate the model output text data from the input text data.
14 . A method comprising:
obtaining input text data and associated output text data;
determining a sentence count of the output text data, wherein the sentence count indicates a number of sentences within the output text data;
labeling the output text data with a sentence count label and a sentence number label using the determined sentence count; and
iteratively training a text processing machine learning (ML) model over multiple training iterations based on training data that includes the input text data, the output text data, and the sentence count, wherein iteratively training the text processing ML model includes, during each of the multiple training iterations, training the text processing ML model to generate model output text data from the input text data based on the sentence count, determining loss data based on a difference between the model output text data and the output text data, and adjusting weight values of the text processing ML model using the determined loss data.
15 . The method of claim 14 , further comprising:
iteratively training a sentence count prediction ML model over the multiple training iterations based on the training data, wherein iteratively training the sentence count prediction ML model includes, during each of the multiple training iterations, training the sentence count prediction ML model to predict a sentence count of the model output text data generated by the text processing ML model, determining sentence count prediction loss data based on a difference between the predicted sentence count and an actual sentence count of the model output text data generated by the text processing ML model, and adjusting weight values of the sentence count prediction ML model using the determined sentence count prediction loss data.
16 . The method of claim 15 , wherein the text processing ML model and the sentence count prediction ML model share an encoder layer, and
wherein, during each of the multiple training iterations, the text processing ML model uses the encoder layer to generate the model output text data and the sentence count prediction ML model uses the encoder layer to predict the sentence count of the model output text data generated by the text processing ML model.
17 . The method of claim 15 , wherein the weight values of the text processing ML model and the weight values of the sentence count prediction ML model are adjusted in parallel.
18 . The method of claim 14 , wherein the output text data both translates and summarizes the input text data such that the output text data is in a different language and has a shorter length than the input text data.