Processing structured and unstructured text to identify sensitive information
This disclosure describes techniques that include identifying sensitive information from any appropriate set of data, such as data produced by operations of a business or organization. In one example, this disclosure describes a method that includes receiving text data containing sensitive information, including structured sensitive information and unstructured sensitive information; applying a rule-based model to identify the structured sensitive information in the text data; applying a machine learning model to identify the unstructured sensitive information in the text data, wherein the machine learning model has been trained to identify unstructured sensitive information in text; and generating output text data from the text data by modifying the structured sensitive information identified by the rule-based model and the unstructured sensitive information identified by the machine learning model.
1. A method comprising:
training, by a computing system, a machine learning model to identify unstructured sensitive information in text that has already been scrubbed for structured sensitive information, each instance of structured sensitive information including both a value and a type,
wherein training the machine learning includes training the machine learning model using text data that has already been scrubbed for structured sensitive information, but that retains the type of structured sensitive information, and
wherein training the machine learning model includes tuning the machine learning model, during training, to use information about where the structured sensitive information occurs within the text data to identify instances of unstructured sensitive information;
receiving, by the computing system, text data containing sensitive information, including structured sensitive information and unstructured sensitive information;
applying, by the computing system, a rule-based model to identify the structured sensitive information in the text data;
generating, by the computing system, results from the rule-based model by modifying the structured sensitive information to hide the value of the structured sensitive information but retain the type of the structured sensitive information;
applying, by the computing system, the machine learning model to identify the unstructured sensitive information in the text data by applying the machine learning model to the results from the rule-based model;
generating, by the computing system, output text data from the results from the rule-based model by modifying the unstructured sensitive information identified by the machine learning model;
performing, by the computing system and using the output text data, analytical studies about customer requests; and
training a chat bot using the output text data.
2. The method of claim 1 , further comprising:
publishing the output text data for use in analytical studies about organizational operations.
3. The method of claim 1 ,
wherein the sensitive information includes at least one of personally identifiable information and payment card industry information.
4. The method of claim 1 ,
wherein the rule-based model is a regular expression model configured to identify social security numbers and phone numbers.
5. The method of claim 1 ,
wherein the machine learning model is a neural network configured to identify names and addresses.
6. The method of claim 1 ,
wherein the machine learning model is a neural network probabilistic parser configured to identify names and addresses using Conditional Random Field methods.
7. The method of claim 1 ,
wherein the machine learning model is neural network trained using text transcripts having labeled unstructured sensitive information.
8. A computing system having a storage system and processing circuitry, wherein the processing circuitry has access to the storage system and is configured to perform operations comprising:
training a machine learning model to identify unstructured sensitive information in text that has already been scrubbed for structured sensitive information, each instance of structured sensitive information including both a value and a type,
wherein training the machine learning includes training the machine learning model using text data that has already been scrubbed for structured sensitive information, but that retains the type of structured sensitive information, and
wherein training the machine learning model includes tuning the machine learning model, during training, to use information about where the structured sensitive information occurs within the text data to identify instances of unstructured sensitive information;
receiving text data containing sensitive information, including structured sensitive information and unstructured sensitive information;
applying a rule-based model to identify the structured sensitive information in the text data;
generating results from the rule-based model by modifying the structured sensitive information to hide the value of the structured sensitive information but retain the type of the structured sensitive information;
applying a machine learning model to identify the unstructured sensitive information in the text data by applying the machine learning model to the results from the rule-based model; and
generating output text data from the text data by modifying the structured sensitive information identified by the rule-based model and the unstructured sensitive information identified by the machine learning model;
performing, by the computing system and using the output text data, analytical studies about customer requests; and
training a chat bot using the output text data.
9. The computing system of claim 8 , wherein the processing circuitry is further configured to perform operations comprising:
publishing the output text data for use in analytical studies about organizational operations.
10. A non-transitory computer-readable medium comprising instructions for causing processing circuitry to perform operations comprising:
training a machine learning model to identify unstructured sensitive information in text that has already been scrubbed for structured sensitive information, each instance of structured sensitive information including both a value and a type,
wherein training the machine learning includes training the machine learning model using text data that has already been scrubbed for structured sensitive information, but that retains the type of structured sensitive information, and
wherein training the machine learning model includes tuning the machine learning model, during training, to use information about where the structured sensitive information occurs within the text data to identify instances of unstructured sensitive information;
receiving text data containing sensitive information, including structured sensitive information and unstructured sensitive information;
applying a rule-based model to identify the structured sensitive information in the text data;
generating results from the rule-based model by modifying the structured sensitive information to hide the value of the structured sensitive information but retain the type of the structured sensitive information;
applying a machine learning model to identify the unstructured sensitive information in the text data by applying the machine learning model to the results from the rule-based model;
generating output text data from the text data by modifying the structured sensitive information identified by the rule-based model and the unstructured sensitive information identified by the machine learning model;
performing, by the computing system and using the output text data, analytical studies about customer requests; and
training a chat bot using the output text data.