Systems and methods for automated text labeling
View Patent ↗Methods and systems are provided for labeling text data using one or more machine learning (ML) models. In one embodiment, a method for training an ML model to label text data comprises manually labeling one or more words in a portion of a first set of text data as instances of a predefined entity of interest; extracting one or more example phrases from the labeled portion of text data, submitting an instruction to a Large Language Model (LLM) to label instances of the predefined entity of interest in the first set of text data, the instruction including the one or more example phrases; and training an ML model to label instances of the predefined entity of interest in a second set of text data, using training data including labeled text data of the first set of text data, the labeled text data outputted by the LLM.
1. A method for labeling text data, the method comprising:
manually labeling one or more words in a portion of a first set of text data as instances of a predefined entity of interest;
extracting one or more example phrases from the labeled portion of text data, each example phrase of the one or more example phrases including at least one labeled instance of the predefined entity of interest;
submitting an instruction to a Large Language Model (LLM) to label instances of the predefined entity of interest in the first set of text data, the instruction including the one or more example phrases;
training a machine learning (ML) model to label instances of the predefined entity of interest in a second set of text data, using training data including the labeled text data of the first set of text data, the labeled text data outputted by the LLM.
2. The method of claim 1 , further comprising performing the manual labeling the one or more words, the extracting of the one or more example phrases, the submitting the instruction to the LLM, and the training the ML model using a plurality of graphical user interfaces (GUI) of a single, integrated text annotation system.
3. The method of claim 2 , further comprising pre-annotating the first set of text data or the portion of the first set of text data based on one or more rule-based algorithms, the one or more rule-based algorithms customizable and/or configurable by a user of the text annotation system.
4. The method of claim 3 , wherein a rule-based algorithm of the one or more rule-based algorithms changes an instance of the entity of interest based on the instance matching a pattern described by a regular expression defined in the text annotation system.
5. The method of claim 2 , further comprising manually annotating the labeled text data prior to training the ML model using a manual annotation GUI of the text annotation system.
6. The method of claim 5 , wherein manually annotating the labeled text further comprises:
distributing a plurality of copies of the labeled text to a team of manual annotators via the text annotation system;
receiving annotated text data from each member of the team of manual annotators;
measuring an agreement between annotations made by each member of the team of manual annotators, and displaying the agreement in an inter-annotator agreement GUI of the text annotation system;
resolving conflicts in labels assigned by each member of the team of manual annotators, via a conflict resolution GUI of the text annotation system;
in response to the agreement exceeding a threshold agreement, storing the annotated text data on the text annotation system for training the ML model.
7. The method of claim 5 , wherein the text data is generated by the text annotation system from an image using optical character recognition (OCR), and manually annotating the labeled text data using the manual annotation GUI further comprises:
displaying the image in a reference pane of the manual annotation GUI;
receiving a user selection of a first area of the image via a cursor of the manual annotation GUI;
detecting a first portion of text positioned within the first area;
highlighting a second portion of the labeled text data corresponding to the first portion of text, in an editing pane of the manual annotation GUI;
labeling the highlighted second portion of the labeled text as an instance of a first entity of interest, the first entity of interest selected by a user via a first control element of the manual annotation GUI.
8. The method of claim 7 , further comprising:
receiving a user selection of a second area of the image via the cursor of the manual annotation GUI, the second area located within the first area;
detecting a third portion of text positioned within the second area;
highlighting a fourth portion of the labeled text data corresponding to the third portion of text in the editing pane of the manual annotation GUI;
labeling the highlighted fourth portion of the labeled text as an instance of a second entity of interest, the second entity of interest selected by the user via a second control element of the manual annotation GUI.
9. The method of claim 7 , wherein labeling the highlighted second portion of the labeled text as an instance of a first entity of interest further comprises:
removing an existing label for the highlighted second portion of the labeled text, the existing label selected by a user via a corresponding control element of the manual annotation GUI; and
labeling the highlighted second portion of the labeled text as an instance of the first entity of interest, the first entity of interest selected by a user via the first control element of the manual annotation GUI.