Hierarchical named entity recognition with multi-task setup
A novel system is described for performing hierarchical named entity recognition (“HNER”) processing that includes identifying categories at different hierarchical levels for a named entity. The HNER system uses a novel architecture comprising an encoder model and a system of trained machine learning (ML) models to perform the HNER processing, where each trained model in the system of ML models corresponds to a particular hierarchical level, and each model is trained to extract one or more named entities and predict a category for each extracted named entity for the corresponding hierarchical level. Novel techniques are also described for training the various models in HNER system including an encoder model and models in the system of models.
1 . A method comprising:
providing a text portion as input to a hierarchical named entity recognition (“HNER”) system comprising a plurality of hierarchical level models corresponding to different hierarchical levels of a hierarchy, the plurality of hierarchical level models including a first hierarchical level model corresponding to a first hierarchical level and a second hierarchical level model corresponding to a second hierarchical level, wherein each hierarchical level model corresponding to a hierarchical level is trained to output, for a named entity, a category from a plurality of categories corresponding to the hierarchical level;
identifying, using the first hierarchical level model, a first category for the first hierarchical level for a first named entity extracted from the text portion, wherein the first category is selected from a first plurality of categories that the first hierarchical level model is trained to predict, the first plurality of categories corresponding to the first hierarchical level;
identifying, using the second hierarchical level model, a second category for the second hierarchical level for a second named entity extracted from the text portion, wherein the second category is selected from a second plurality of categories that the second hierarchical level is trained to predict, the second plurality of categories corresponding to the second hierarchical level;
wherein the identifying the first category using the first hierarchical level model is performed independently of the identifying the second category using the second hierarchical level model;
computing, for each hierarchical level in the hierarchy, a hierarchical level loss for the hierarchical level based upon the output generated by a hierarchical level model from the plurality of hierarchical level models corresponding to the hierarchical level and ground truth information for the hierarchical level identified from annotation information associated with at least a first training datapoint in a plurality of training datapoints used to train the plurality of hierarchical level models; and
updating the plurality of hierarchical level models based at least in part on the hierarchical level loss computed for each hierarchical level in the hierarchy, wherein updating the plurality of hierarchical level models comprises updating one or more trainable parameters associated with the plurality of hierarchical level models to minimize an aggregate loss for the HNER system.
2 . The method of claim 1 , wherein the first named entity and the second named entity are the same named entity.
3 . The method of claim 1 , further comprising identifying, using the first hierarchical level model, a third category for the first hierarchical level for a third named entity extracted from the text portion, wherein the third category is selected from a third plurality of categories that the first hierarchical level model is trained to predict, the third plurality of categories corresponding to the first hierarchical level.
4 . The method of claim 1 , wherein identifying the first category using the first hierarchical level model is performed in parallel with identifying the second category using the second hierarchical level model.
5 . The method of claim 1 , further comprising outputting information indicative of the first named entity, the first category, the second named entity, and the second category.
6 . The method of claim 1 , further comprising:
generating, by an encoder of the HNER system, an encoded representation of the text portion; and
providing the encoded representation to both the first hierarchical level model and to the second hierarchical level model.
7 . The method of claim 6 , wherein the encoder is a Bidirectional Encoder Representations from Transformers (“BERT”) model.
8 . The method of claim 6 , wherein the providing comprises providing the encoded representation to the first hierarchical level model and the second hierarchical level model in parallel.
9 . The method of claim 6 , further comprising training the encoder and the plurality of hierarchical level models using the plurality of training data points.
10 . The method of claim 9 , wherein training the plurality of hierarchical level models comprises:
training the first hierarchical level model to predict categories in the first plurality of categories corresponding to the first hierarchical level; and
training the second hierarchical level model to predict categories in the second plurality of categories corresponding to the second hierarchical level.
11 . The method of claim 10 , wherein training the encoder and the plurality of hierarchical level models comprises:
for the first training datapoint in the plurality of training datapoints, providing a text portion of the first training datapoint to the encoder;
based upon the annotation information associated with the first training datapoint, identifying, for each hierarchical level in the hierarchy, the ground truth information for the hierarchical level, wherein the ground truth information for the hierarchical level comprises information indicative of a named entity in the text portion of the first training datapoint and a category for the named entity, wherein the category is one of a plurality of categories corresponding to the hierarchical level;
generating by the encoder an encoded representation for the text portion of the first training datapoint;
providing the encoded representation for the text portion of the first training datapoint to each hierarchical level in the plurality of hierarchical level models; and
generating, by each hierarchical level model in the plurality of hierarchical level models, a prediction for the text portion of the first training datapoint, wherein the prediction by a hierarchical level model identifies a named entity extracted by the hierarchical level model from the text portion of the first training datapoint for that hierarchical level corresponding to the hierarchical level model and a category predicted by the hierarchical level model for the extracted named entity, wherein the category predicted by the hierarchical level model is from a plurality of categories for the hierarchical level.
12 . The method of claim 11 , wherein training the encoder and the plurality of hierarchical level models further comprises:
generating the aggregate loss based upon the hierarchical level losses computed for a plurality of hierarchical levels; and
minimizing the aggregate loss using a loss minimization technique, wherein the minimizing comprises updating the encoder and the plurality of hierarchical level models.
13 . The method of claim 12 , wherein updating the encoder and the plurality of hierarchical level models further comprises updating one or more trainable parameters associated with the encoder and the plurality of hierarchical level models.
14 . The method of claim 12 , wherein the loss minimization technique is cross-entropy loss function.
15 . A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
providing a text portion as input to a hierarchical named entity recognition (“HNER”) system comprising a plurality of hierarchical level models corresponding to different hierarchical levels of a hierarchy, the plurality of hierarchical level models including a first hierarchical level model corresponding to a first hierarchical level and a second hierarchical level model corresponding to a second hierarchical level, wherein each hierarchical level model corresponding to a hierarchical level is trained to output, for a named entity, a category from a plurality of categories corresponding to the hierarchical level;
identifying, using the first hierarchical level model, a first category for the first hierarchical level for a first named entity extracted from the text portion, wherein the first category is selected from a first plurality of categories that the first hierarchical level model is trained to predict, the first plurality of categories corresponding to the first hierarchical level; and
identifying, using the second hierarchical level model, a second category for the second hierarchical level for a second named entity extracted from the text portion, wherein the second category is selected from a second plurality of categories that the second hierarchical level is trained to predict, the second plurality of categories corresponding to the second hierarchical level,
wherein the identifying the first category using the first hierarchical level model is performed independently of and in parallel with the identifying the second category using the second hierarchical level model;
computing, for each hierarchical level in the hierarchy, a hierarchical level loss for the hierarchical level based upon the output generated by a hierarchical level model from the plurality of hierarchical level models corresponding to the hierarchical level and ground truth information for the hierarchical level identified from annotation information associated with at least a first training datapoint in a plurality of training datapoints used to train the plurality of hierarchical level models; and
updating the plurality of hierarchical level models based at least in part on the hierarchical level loss computed for each hierarchical level in the hierarchy, wherein updating the plurality of hierarchical level models comprises updating one or more trainable parameters associated with the plurality of hierarchical level models to minimize an aggregate loss for the HNER system.
16 . The non-transitory computer-readable storage medium of claim 15 , further comprising:
generating, by an the encoder of the HNER system, an encoded representation of the text portion; and
providing the encoded representation to both the first hierarchical level model and to the second hierarchical level model in parallel.
17 . The non-transitory computer-readable storage medium of claim 16 , further comprising:
training the encoder and the plurality of hierarchical level models using the plurality of training datapoints;
for the first training datapoint in the plurality of training datapoints, providing a text portion of the first training datapoint to the encoder;
based upon the annotation information associated with the first training datapoint, identifying, for each hierarchical level in the hierarchy, the ground truth information for the hierarchical level, where the ground truth information for the hierarchical level comprises information indicative of a named entity in the text portion of the first training datapoint and a category for the named entity, wherein the category is one of a plurality of categories corresponding to the hierarchical level;
generating by the encoder an encoded representation for the text portion of the first training datapoint;
providing the encoded representation for the text portion of the first training datapoint to each hierarchical level in the plurality of hierarchical level models; and
generating, by each hierarchical level model in the plurality of hierarchical level models, a prediction for the text portion of the first training datapoint, where the prediction by a hierarchical level model identifies a named entity extracted by the hierarchical level model from the text portion of the first training datapoint for that hierarchical level corresponding to the hierarchical level model and a category predicted by the hierarchical level model for the extracted named entity, wherein the category predicted by the hierarchical level model is from a plurality of categories for the hierarchical level.
18 . A system, comprising:
one or more processors; and
one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform:
providing a text portion as input to a hierarchical named entity recognition (“HNER”) system comprising a plurality of hierarchical level models corresponding to different hierarchical levels of a hierarchy, the plurality of hierarchical level models including a first hierarchical level model corresponding to a first hierarchical level and a second hierarchical level model corresponding to a second hierarchical level, wherein each hierarchical level model corresponding to a hierarchical level is trained to output, for a named entity, a category from a plurality of categories corresponding to the hierarchical level;
identifying, using the first hierarchical level model, a first category for the first hierarchical level for a first named entity extracted from the text portion, wherein the first category is selected from a first plurality of categories that the first hierarchical level model is trained to predict, the first plurality of categories corresponding to the first hierarchical level; and
identifying, using the second hierarchical level model, a second category for the second hierarchical level for a second named entity extracted from the text portion, wherein the second category is selected from a second plurality of categories that the second hierarchical level is trained to predict, the second plurality of categories corresponding to the second hierarchical level,
wherein the identifying the first category using the first hierarchical level model is performed independently of and in parallel with the identifying the second category using the second hierarchical level model;
computing, for each hierarchical level in the hierarchy, a hierarchical level loss for the hierarchical level based upon the output generated by a hierarchical level model from the plurality of hierarchical level models corresponding to the hierarchical level and ground truth information for the hierarchical level identified from annotation information associated with at least a first training datapoint in a plurality of training datapoints used to train the plurality of hierarchical level models; and
updating the plurality of hierarchical level models based at least in part on the hierarchical level loss computed for each hierarchical level in the hierarchy, wherein updating the plurality of hierarchical level models comprises updating one or more trainable parameters associated with the plurality of hierarchical level models to minimize an aggregate loss for the HNER system.
19 . The system of claim 18 , further comprising:
generating, by an encoder of the HNER system, an encoded representation of the text portion; and
providing the encoded representation to both the first hierarchical level model and to the second hierarchical level model in parallel.
20 . The system of claim 19 , further comprising:
training the encoder and the plurality of hierarchical level models using the plurality of training datapoints;
for the first training datapoint in the plurality of training datapoints, providing a text portion of the first training datapoint to the encoder;
based upon the annotation information associated with the first training datapoint, identifying, for each hierarchical level in the hierarchy, the ground truth information for the hierarchical level, where the ground truth information for the hierarchical level comprises information indicative of a named entity in the text portion of the first training datapoint and a category for the named entity, wherein the category is one of a plurality of categories corresponding to the hierarchical level;
generating by the encoder an encoded representation for the text portion of the first training datapoint;
providing the encoded representation for the text portion of the first training datapoint to each hierarchical level in the plurality of hierarchical level models; and
generating, by each hierarchical level model in the plurality of hierarchical level models, a prediction for the text portion of the first training datapoint, where the prediction by a hierarchical level model identifies a named entity extracted by the hierarchical level model from the text portion of the first training datapoint for that hierarchical level corresponding to the hierarchical level model and a category predicted by the hierarchical level model for the extracted named entity, wherein the category predicted by the hierarchical level model is from a plurality of categories for the hierarchical level.