Implementing active learning in natural language generation tasks
Methods, systems, and computer program products for implementing active learning in NLG tasks are provided herein. A computer-implemented method includes generating multiple natural language annotations associated with multiple items of unlabeled data by processing the unlabeled data using at least one artificial intelligence model; determining at least one quality score attributed to at least a portion of the multiple generated natural language annotations based at least in part on at least one quality metric; selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data based at least in part on the at least one determined quality score; and performing one or more automated actions based at least in part on the at least one selected natural language annotation.
1 . A system comprising:
a memory configured to store program instructions; and
a processor operatively coupled to the memory to execute the program instructions to:
generate multiple natural language annotations associated with multiple items of unlabeled data by processing the unlabeled data using at least one artificial intelligence model;
determine at least one quality score attributed to at least a portion of the multiple generated natural language annotations based at least in part on at least one quality metric;
select at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data based at least in part on the at least one determined quality score, wherein selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data comprises selecting the at least one of the multiple natural language annotations and the at least one corresponding item of the multiple items of unlabeled data based at least in part on at least one difference between the at least one determined quality score and one or more predicted quality scores pertaining to one or more manual annotations associated with the multiple items of unlabeled data; and
perform one or more automated actions based at least in part on the at least one selected natural language annotation, wherein performing one or more automated actions comprises automatically training, using the least one selected natural language annotation and the at least one corresponding item of unlabeled data, the at least one artificial intelligence model to be used in connection with one or more natural language generation (NLG) tasks.
2 . The system of claim 1 , wherein performing one or more automated actions comprises automatically training one or more additional artificial intelligence models to be used in connection with one or more NLG tasks using the least one selected natural language annotation and the at least one corresponding item of unlabeled data.
3 . The system of claim 1 , wherein processing the unlabeled data comprises processing the unlabeled data using at least one artificial intelligence-based natural language generation model.
4 . The system of claim 3 , wherein the at least one artificial intelligence-based natural language generation model comprises at least one transformer-based generative model.
5 . The system of claim 4 , wherein the at least one transformer-based generative model comprises at least one of one or more encoder-decoder models and one or more denoising autoencoders.
6 . The system of claim 1 , wherein generating the multiple natural language annotations comprises processing the unlabeled data using at least one artificial intelligence model in accordance with one or more predetermined characteristics of a given NLG task.
7 . The system of claim 1 , wherein determining at least one quality score comprises determining the at least one quality score attributed to the at least a portion of the multiple generated natural language annotations based at least in part on one or more reference-less evaluation metrics.
8 . The system of claim 1 , wherein selecting at least one of the multiple natural language annotations comprises selecting the at least one natural language annotation which corresponds with improvement associated with the at least one artificial intelligence model, as measured by the at least one quality metric.
9 . The system of claim 1 , wherein the processor is further operatively coupled to the memory to execute the program instructions to:
predict the one or more quality scores pertaining to the one or more manual annotations associated with the multiple items of unlabeled data.
10 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:
generate multiple natural language annotations associated with multiple items of unlabeled data by processing the unlabeled data using at least one artificial intelligence model;
determine at least one quality score attributed to at least a portion of the multiple generated natural language annotations based at least in part on at least one quality metric;
select at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data based at least in part on the at least one determined quality score, wherein selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data comprises selecting the at least one of the multiple natural language annotations and the at least one corresponding item of the multiple items of unlabeled data based at least in part on at least one difference between the at least one determined quality score and one or more predicted quality scores pertaining to one or more manual annotations associated with the multiple items of unlabeled data; and
perform one or more automated actions based at least in part on the at least one selected natural language annotation, wherein performing one or more automated actions comprises automatically training, using the least one selected natural language annotation and the at least one corresponding item of unlabeled data, the at least one artificial intelligence model to be used in connection with one or more natural language generation (NLG) tasks.
11 . The computer program product of claim 10 , wherein processing the unlabeled data comprises processing the unlabeled data using at least one artificial intelligence-based natural language generation model.
12 . The computer program product of claim 10 , wherein the program instructions executable by a computing device further cause the computing device to:
predict the one or more quality scores pertaining to the one or more manual annotations associated with the multiple items of unlabeled data.
13 . A computer-implemented method comprising:
generating multiple natural language annotations associated with multiple items of unlabeled data by processing the unlabeled data using at least one artificial intelligence model;
determining at least one quality score attributed to at least a portion of the multiple generated natural language annotations based at least in part on at least one quality metric;
selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data based at least in part on the at least one determined quality score, wherein selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data comprises selecting the at least one of the multiple natural language annotations and the at least one corresponding item of the multiple items of unlabeled data based at least in part on at least one difference between the at least one determined quality score and one or more predicted quality scores pertaining to one or more manual annotations associated with the multiple items of unlabeled data; and
performing one or more automated actions based at least in part on the at least one selected natural language annotation, wherein performing one or more automated actions comprises automatically training, using the least one selected natural language annotation and the at least one corresponding item of unlabeled data, the at least one artificial intelligence model to be used in connection with one or more natural language generation (NLG) tasks;
wherein the method is carried out by at least one computing device.
14 . The computer-implemented method of claim 13 , further comprising:
predicting the one or more quality scores pertaining to the one or more manual annotations associated with the multiple items of unlabeled data.
15 . The computer-implemented method of claim 13 , wherein software implementing the method is provided as a service in a cloud environment.
16 . The computer-implemented method of claim 13 , wherein processing the unlabeled data comprises processing the unlabeled data using at least one artificial intelligence-based natural language generation model.
17 . The computer-implemented method of claim 13 , wherein generating the multiple natural language annotations comprises processing the unlabeled data using at least one artificial intelligence model in accordance with one or more predetermined characteristics of a given NLG task.
18 . The computer-implemented method of claim 13 , wherein determining at least one quality score comprises determining the at least one quality score attributed to the at least a portion of the multiple generated natural language annotations based at least in part on one or more reference-less evaluation metrics.
19 . The computer-implemented method of claim 13 , wherein performing one or more automated actions comprises automatically training one or more additional artificial intelligence models to be used in connection with one or NLG tasks using the least one selected natural language annotation and the at least one corresponding item of unlabeled data.
20 . The computer-implemented method of claim 13 , wherein selecting at least one of the multiple natural language annotations comprises selecting the at least one natural language annotation which corresponds with improvement associated with the at least one artificial intelligence model, as measured by the at least one quality metric.