Tagging performance evaluation and improvement
According to one implementation, a tagging performance evaluation system includes a computing platform having a hardware processor and a memory storing a software code. The hardware processor is configured to execute the software code to receive annotation data identifying content, annotation tags applied to the content, and one or more correction(s) to the annotation tags, to perform, using the annotation data, at least one of an evaluation of a tagging process resulting in application of the annotation tags to the content or an assessment of a correction process resulting in the correction(s), and to identify, based on the at least one of the evaluation or the assessment, one or more parameters for improving at least one of the tagging process or the correction process. At least one of the evaluation or the assessment is performed using a machine learning model of the tagging performance evaluation system.
1 . A system comprising:
a computing platform including a hardware processor and a system memory storing a software code, an annotation evaluation machine learning model, and a correction assessment machine learning model;
the hardware processor configured to execute the software code to:
receive annotation data identifying a content and a plurality of annotation tags applied to the content in a tagging process performed by a tagging entity;
receive one or more corrections to the plurality of annotation tags, the one or more corrections having been made in a correction process performed by a quality assurance (QA) entity, wherein at least one of (i) the tagging entity is a trained tagging machine learning model communicatively coupled to the system via a communication network, or (ii) the QA entity is a trained tag review and correction machine learning model communicatively coupled to the system via the communication network;
perform, using the annotation evaluation machine learning model and the annotation data, an automated evaluation of the tagging process based on the plurality of annotation tags and the one or more corrections to the plurality of annotation tags;
perform, using the correction assessment machine learning model, the automated evaluation of the tagging process and the annotation data based at least in part on how many corrections are included among the one or more corrections to the plurality of annotation tags;
identify, based on at least one of the automated evaluation of the tagging process or the automated assessment of the correction process, one or more parameters for improving at least one of the tagging process or the correction process; and
modify, based on the one or more parameters, one or more of stored weights or stored priorities of at least one of the trained tagging machine learning model communicatively coupled to the system or the trained tag review and correction machine learning model communicatively coupled to the system, wherein the one or more of the stored weights or stored priorities are updated using the one or more parameters as training feedback, the one or more parameters including at least one of a tagging performance history of the tagging entity or a correction history of the QA entity, to provide at least one of an improved tagging machine learning model or an improved tag review and correction machine learning model.
2 . The system of claim 1 , wherein the tagging entity is the trained tagging machine learning model and the QA entity is the trained tag review and correction machine learning model.
3 . The system of claim 1 , wherein the trained tagging machine learning model and the trained tag review and correction machine learning model are further improved by modifying a predetermined taxonomy of tags.
4 . The system of claim 1 , wherein the annotation evaluation machine learning model includes a Support Vector Machine (SVM).
5 . The system of claim 1 , wherein the hardware processor is further configured to execute the software code to:
produce one or more reports based on the one or more parameters; and
output the one or more reports to at least one of the tagging entity performing the tagging process, the QA entity performing the correction process, or an annotation administrator.
6 . The system of claim 5 , wherein the tagging entity is a human tagger, and wherein one of the one or more reports is delivered to the human tagger when the human tagger begins a next tagging process.
7 . The system of claim 5 , wherein the QA entity is a human QA reviewer, and wherein one of the one or more reports is delivered to the human QA reviewer before the human reviewer begins a next correction process.
8 . A method for use by a system including a computing platform having a hardware processor and a system memory storing a software code, an annotation evaluation machine learning model and a correction assessment machine learning model, the method comprising:
receiving, by the software code executed by the hardware processor, annotation data identifying a content and a plurality of annotation tags applied to the content in a tagging process performed by a tagging entity;
receiving, by the software code executed by the hardware processor, one or more corrections to the plurality of annotation tags, the one or more corrections having been made in a correction process performed by a quality assurance (QA) entity, wherein at least one of (i) the tagging entity is a trained tagging machine learning model communicatively coupled to the system via a communication network, or (ii) the QA entity is a trained tag review and correction machine learning model communicatively coupled to the system via the communication network;
performing, by the software code executed by the hardware processor and using the annotation evaluation machine learning model and the annotation data, an automated evaluation of the tagging process based on the plurality of annotation tags and the one or more corrections to the plurality of annotation tags;
performing, by the software code executed by the hardware processor and using the correction assessment machine learning model, the automated evaluation of the tagging process and the annotation data based at least in part on how many corrections are included among the one or more corrections to the plurality of annotation tags;
identifying, by the software code executed by the hardware processor and based on at least one of the automated evaluation of the tagging process or the automated assessment of the correction process, one or more parameters for improving at least one of the tagging process or the correction process; and
modifying, by the software code executed by the hardware processor and based on the one or more parameters, one or more of stored weights or stored priorities of at least one of the trained tagging machine learning model communicatively coupled to the system or the trained tag review and correction machine learning model communicatively coupled to the system, wherein the one or more of the stored weights or stored priorities are updated using the one or more parameters as training feedback, the one or more parameters including at least one of a tagging performance history of the tagging entity or a correction history of the QA entity, to provide at least one of an improved tagging machine learning model or an improved tag review and correction machine learning model.
9 . The method of claim 8 , wherein the tagging entity is the trained tagging machine learning model and the QA entity is the trained tag review and correction machine learning model.
10 . The method of claim 8 , wherein the trained tagging machine learning model and the trained tag review and correction machine learning model are further improved by modifying a predetermined taxonomy of tags.
11 . The method of claim 8 , wherein the annotation evaluation machine learning model includes a Support Vector Machine (SVM).
12 . The method of claim 8 , further comprising:
producing, by the software code executed by the hardware processor, one or more reports based on the one or more parameters; and
outputting, by the software code executed by the hardware processor, the one or more reports to at least one of the tagging entity performing the tagging process, the QA entity performing the correction process, or an annotation administrator.
13 . The method of claim 12 , wherein the tagging entity is a human tagger, and wherein one of the one or more reports is delivered to the human tagger when the human tagger begins a next tagging process.
14 . The method of claim 12 , wherein the QA entity is a human QA reviewer, and wherein one of the one or more reports is delivered to the human QA reviewer before the human reviewer begins a next correction process.