Method of learning task model and language model, and learning device for performing the same
A method of learning a task model and a language model according to an embodiment of the present application includes: acquiring log data for each user for a content set consumed by a plurality of users and content included in the content set; acquiring text embedding for the content included in the content set through a pretrained language model (PLM); acquiring an output value related to a task from the text embedding and the log data through a task model; updating a weight of the task model based on the output value and a target value of the task; and acquiring back-propagation information for training the language model from the task model, and training the language model based on the back-propagation information.
1 . A method of learning a task model and a pretrained language model (PLM) by a learning device for tuning the task model using the PLM, the method comprising:
acquiring log data of a plurality of users for contents included in a content set consumed by the plurality of users and the contents included in the content set, the plurality of users including a first user and a second user, and the contents including a first content and a second content;
acquiring text embedding for the contents included in the content set through the PLM;
acquiring an output value related to a task of the task model from the text embedding and the log data through the task model;
updating a weight of the task model based on the output value and a target value of the task; and
acquiring back-propagation information for training the PLM from the task model and training the PLM based on the back-propagation information,
wherein the method further comprises:
acquiring execution data for executing the PLM, the executing data including at least one of layer information, computation information and weight information of the PLM, and
acquiring the PLM based on the execution data, the PLM being one of a Bidirectional Encoder Representations from Transformers (BERT) model, a Generative Pre-trained Transformer 3 (GPT-3) model, a Robustly Optimized BERT Pre-Training Approach (ROBERTA) model, a Unified pre-trained Language Model (UniLM), and a Minimal Language Model (MiniLM),
wherein the training of the PLM includes:
classifying the log data of the plurality of users as a plurality of log data sets according to each content of the contents, wherein the classifying of the log data of the plurality of users includes (i) classifying log data of the first user for the first content and log data of the second user for the first content as a first log data set of the plurality of log data sets, and (ii) classifying log data of the first user for the second content and log data of the second user for the second content as a second log data set of the plurality of log data sets, and
training the PLM based on back-propagation information corresponding to each content of the contents by inputting each of the plurality of classified log data sets to the PLM to increase accuracy of the task model,
wherein a total number of the classified log data sets is the same as a total number of the contents, and
wherein the training of the PLM based on the back-propagation information corresponding to each content of the contents is performed as many as the total number of the contents.
2 . The method of claim 1 , wherein the training of the PLM further includes updating a parameter included in the PLM based on the back-propagation information corresponding to each content of the contents.
3 . The method of claim 1 , wherein time complexity required to train the PLM is related to the total number of the contents included in the content set.
4 . A non-transitory computer-readable recording medium in which a computer program executed by a computer is recorded, the computer program comprising:
acquiring log data of a plurality of users for contents included in a content set consumed by the plurality of users and the contents included in the content set, the plurality of users including a first user and a second user, and the contents including a first content and a second content;
acquiring text embedding for the contents included in the content set through a pretrained language model (PLM);
acquiring an output value related to a task of a task model from the text embedding and the log data through the task model;
updating a weight of the task model based on the output value and a target value of the task; and
acquiring back-propagation information for training the PLM from the task model and training the PLM based on the back-propagation information,
wherein the computer program further comprises:
acquiring execution data for executing the PLM, the executing data including at least one of layer information, computation information and weight information of the PLM, and
acquiring the PLM based on the execution data, the PLM being one of a Bidirectional Encoder Representations from Transformers (BERT) model, a Generative Pre-trained Transformer 3 (GPT-3) model, a Robustly Optimized BERT Pre-Training Approach (ROBERTA) model, a Unified pre-trained Language Model (UniLM), and a Minimal Language Model (MiniLM),
wherein the training of the PLM includes:
classifying the log data of the plurality of users as a plurality of log data sets according to each content of the contents, wherein the classifying of the log data of the plurality of users includes (i) classifying log data of the first user for the first content and log data of the second user for the first content as a first log data set of the plurality of log data sets, and (ii) classifying log data of the first user for the second content and log data of the second user for the second content as a second log data set of the plurality of log data sets, and
training the PLM based on back-propagation information corresponding to each content of the contents by inputting each of the plurality of classified log data sets to the PLM to increase accuracy of the task model,
wherein a total number of the classified log data sets is the same as a total number of the contents, and
wherein the training of the PLM based on the back-propagation information corresponding to each content of the contents is performed as many as the total number of the contents.
5 . A learning device for tuning a task model using a pretrained language model (PLM), the learning device comprising:
a transceiver configured to transmit/receive a content set including a plurality of pieces of contents and log data of a plurality of users for the contents included in the content set; and
a controller configured to train the PLM and the task model based on the contents and the log data,
wherein the controller is configured to:
acquire the log data of the plurality of users for the contents included in the content set consumed by the plurality of users and the contents included in the content set, the plurality of users including a first user and a second user, and the contents including a first content and a second content,
acquire text embedding for the contents included in the content set through the PLM,
acquire an output value related to a task of the task model from the text embedding and the log data through the task model,
update a weight of the task model based on the output value and the target value of the task, and
acquire back-propagation information for training the PLM from the task model, and train the PLM based on the back-propagation information,
wherein the controller is further configured to:
acquire execution data for executing the PLM, the executing data including at least one of layer information, computation information and weight information of the PLM, and
acquire the PLM based on the execution data, the PLM being one of a Bidirectional Encoder Representations from Transformers (BERT) model, a Generative Pre-trained Transformer 3 (GPT-3) model, a Robustly Optimized BERT Pre-Training Approach (ROBERTA) model, a Unified pre-trained Language Model (UniLM), and a Minimal Language Model (MiniLM),
wherein the training of the PLM includes:
classifying the log data of the plurality of users as a plurality of log data sets according to each content of the contents, wherein the classifying of the log data of the plurality of users includes (i) classifying log data of the first user for the first content and log data of the second user for the first content as a first log data set of the plurality of log data sets, and (ii) classifying log data of the first user for the second content and log data of the second user for the second content as a second log data set of the plurality of log data sets, and
training the PLM based on back-propagation information corresponding to each content of the contents by inputting each of the plurality of classified log data sets to the PLM to increase accuracy of the task model,
wherein a total number of the classified log data sets is the same as a total number of the contents, and
wherein the training of the PLM based on the back-propagation information corresponding to each content of the contents is performed as many as the total number of the contents.