Named-entity recognition of protected health information
A named-entity recognition (NER) model detects named entities with types that correspond to protected health information (PHI) in potentially sensitive documents. The NER model is trained to detect named entities corresponding to both personally identifiable information (PII) and medical terms. Output of the NER model is preprocessed as input to a random forest classifier that outputs a verdict that documents comprise sensitive data. The verdict is interpretable via high confidence named entities detected by the NER model that led to the verdict.
1 . A method comprising:
determining a plurality of named entities in a document and a plurality of confidence values indicating confidence of types for each of the plurality of named entities with a named-entity recognition model, wherein the document comprises potentially sensitive data, wherein the named-entity recognition model was at least partially trained on named entities known to correspond to sensitive data;
identifying at least one of a maximal confidence value and a mean confidence value based on the plurality of confidence values for each type of entity in the plurality of named entities;
concatenating the at least one of the maximal confidence value and the mean confidence value for each type of entity to generate a vector of confidence values;
inputting the vector of confidence values and a plurality of feature values generated from the plurality of named entities into a tree-based classifier to output a verdict indicating whether the document comprises sensitive data;
determining whether the document comprises sensitive data based on the verdict, wherein the output of the tree-based classifier further indicates one or more feature importances for one or more types of the plurality of named entities in determining whether the document comprises sensitive data, wherein each of the one or more feature importances indicates relative importance of a corresponding one of the one or more types of named entities for the verdict; and
based on the verdict indicating that the document comprises sensitive data, identifying, with the plurality of confidence values and the one or more feature importances, one or more of the plurality of named entities as high confidence named entities for the document comprising sensitive data.
2 . The method of claim 1 , wherein the named entities known to correspond to sensitive data comprise entities corresponding to personally identifiable information and medical terms, wherein the potentially sensitive data in the document comprise protected health information.
3 . The method of claim 1 , wherein the named-entity recognition model comprises a Bidirectional Encoder Representations from Transformers model.
4 . A non-transitory machine-readable medium having program code stored thereon, the program code comprising instructions to:
input a plurality of tokens of a document into a named-entity recognition model; obtain, from output of the named-entity recognition model, a plurality of named entities and a plurality of confidence values indicating confidence in types of each of the plurality of named entities, wherein the document comprises potentially sensitive data, wherein the plurality of named entities correspond to a plurality of subsets of the document;
identify at least one of a maximal confidence value and a mean confidence value based on the plurality of confidence values for each type of entity in the plurality of named entities;
concatenate the at least one of the maximal confidence value and the mean confidence value for each type of entity to generate a vector of confidence values;
input the vector of confidence values and a plurality of feature values generated from the plurality of named entities into a tree-based classifier to output a verdict indicating whether the document comprises sensitive data;
determine whether the document comprises sensitive data based on the verdict and determine one or more feature importances for one or more types of the plurality of named entities in the determination of whether the document comprises sensitive data, wherein each of the one or more feature importances indicates relative importance of a corresponding one of the one or more types of named entities for the verdict; and
based on a determination that the document comprises sensitive data indicated in output of the tree-based classifier, identify, with the plurality of confidence values and the one or more feature importances, one or more of the plurality of named entities corresponding to subsets of the document that comprise sensitive data.
5 . The non-transitory machine-readable medium of claim 4 , wherein the plurality of named entities comprise named entities corresponding to personally identifiable information and medical terms, wherein the potentially sensitive data in the document comprise protected health information.
6 . The non-transitory machine-readable medium of claim 4 , wherein the named-entity recognition model comprises a Bidirectional Encoder Representations from Transformers model.
7 . An apparatus comprising:
a processor; and a machine-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to,
identify a plurality of named entities in a document with a plurality of confidence values indicating confidence of types of each of the plurality of named entities based on input of a plurality of tokens of the document into a named-entity recognition model;
identify at least one of a maximal confidence value and a mean confidence value based on the plurality of confidence values for each type of entity in the plurality of named entities;
concatenate the at least one of the maximal confidence value and the mean confidence value for each type of entity to generate a vector of confidence values;
input a plurality of feature values of the of named entities and plurality the vector of confidence values into a tree-based classifier to determine whether the document comprises sensitive data and to determine one or more feature importances for one or more types of the plurality of named entities in the determination of whether the document comprises sensitive data, wherein each of the one or more feature importances indicates relative importance of a corresponding one of the one or more types of named entities for the document comprising sensitive data; and
based on a determination that the document comprises sensitive data, identify, with the plurality of confidence values and the one or more feature importances, one or more of the plurality of named entities as high confidence named entities that the document comprises sensitive data.
8 . The apparatus of claim 7 , wherein the plurality of named entities comprise entities that correspond to personally identifiable information and medical terms, wherein the determination that the document comprises sensitive data comprises a determination that the document comprises protected health information.
9 . The apparatus of claim 7 wherein the named-entity recognition model comprises a Bidirectional Encoder Representations from Transformers model.
10 . The apparatus of claim 7 , wherein the machine-readable medium further has stored thereon instructions executable by the processor to cause the apparatus to generate the plurality of feature values, wherein the instructions executable by the processor to cause the apparatus to generate the plurality of feature values comprise instructions to vectorize the plurality of confidence values according to corresponding ones of the plurality of named entities to generate a vector of confidence values as the plurality of feature values.
11 . The apparatus of claim 7 , wherein the machine-readable medium further has stored thereon instructions executable by the processor to cause the apparatus to indicate decisions at a plurality of nodes in the tree-based classifier that determined whether the document comprises sensitive data.
12 . The apparatus of claim 7 , wherein the instructions executable by the processor to cause the apparatus to identify the plurality of named entities and the plurality of confidence values of the named entities in the document further comprise instructions to: subdivide the document into a plurality of sub-documents; and tokenize the plurality of sub-documents to extract the plurality of tokens.
13 . The method of claim 1 , wherein identifying the one or more of the plurality of named entities as high confidence named entities for the document comprising sensitive data comprises, for a subset of the one or more types having top-N features importances in the one or more feature importances, identifying those of the plurality of named entities having highest confidence values in the plurality of confidence values for indicating one of the subset of the one or more types.
14 . The non-transitory machine-readable medium of claim 4 , wherein the program code to identify the one or more of the plurality of named entities as high confidence named entities for the document comprising sensitive data comprises instructions to, for a subset of the one or more types having top-N features importances in the one or more feature importances, identify those of the plurality of named entities having highest confidence values in the plurality of confidence values for indicating one of the subset of the one or more types.
15 . The apparatus of claim 7 , wherein the instructions to identify the one or more of the plurality of named entities as high confidence named entities for the document comprising sensitive data comprise instructions executable by the processor to cause the apparatus to, for a subset of the one or more types having top-N features importances in the one or more feature importances, identify those of the plurality of named entities having highest confidence values in the plurality of confidence values for indicating one of the subset of the one or more types.
16 . The method of claim 1 , further comprising indicating decisions at a plurality of nodes in the tree-based classifier that determined whether the document comprises sensitive data.
17 . The method of claim 1 , further comprising truncating and tokenizing the document prior to determining the plurality of named entities in the document and the plurality of confidence values.
18 . The method of claim 1 , further comprising indicating a severity level associated with the verdict.
19 . The non-transitory machine-readable medium of claim 4 , wherein the program code further comprises instructions to indicate decisions at a plurality of nodes in the tree-based classifier that determined whether the document comprises sensitive data.
20 . The non-transitory machine-readable medium of claim 4 , wherein the program code further comprises instructions to truncate and tokenize the document prior to the determination of the plurality of named entities in the document and the plurality of confidence values.