Layered masking of content
Methods, systems and computer program products for layered masking of data are described. A system receives content including personally identifiable information (PII). The system redacts the content by masking the PII. The system identifies the PII in multi-layer processing, where in each layer, the system determines a respective confidence score indicating a probability that a token is PII. If the confidence score is sufficiently high, the system masks the token. Otherwise, the system provides the token to a next layer for processing. The layers can include regular expression based processing, lookup table based processing, and machine learning based processing.
1. A method comprising:
storing a plurality of regular expressions for identifying personally identifiable information (PII), wherein each regular expression of the plurality of regular expressions comprises a sequence of symbols and characters expressing a string or pattern;
storing for each regular expression of the plurality of regular expressions a corresponding probability that the regular expression identifies the PII;
receiving content that includes a token;
comparing data within the token with each of the plurality of regular expressions;
determining, based on comparing each of the plurality of regular expressions with the data within the token, whether the data within the token matches a regular expression of the plurality of regular expressions;
based on determining that the data within the token matches the regular expression of the plurality of regular expressions, retrieving a probability associated the regular expression that matches the data within the token;
calculating a confidence score according to the probability associated the regular expression that matches the data within the token; and
masking the token in accordance with the confidence score.
2. The method of claim 1 , wherein masking the token in accordance with the confidence score comprises:
comparing the confidence score to a threshold; and
based on determining that the confidence score is higher than the threshold, masking the token.
3. The method of claim 1 further comprising:
retrieving historical transaction data;
determining, from the historical transaction data, patterns for the PII; and
converting the patterns to the plurality of regular expressions.
4. The method of claim 3 , further comprising generating clustered groups from the historical transaction data.
5. The method of claim 1 wherein:
the content includes transaction data, the transaction data including one or more transaction records each corresponding to a respective transaction and each including a description of the transaction, and
the token is a part of the description and includes at least one of a number or a word.
6. The method of claim 1 , wherein storing a plurality of regular expressions comprises storing the plurality of regular expressions in a database.
7. The method of claim 3 , further comprising deriving the first confidence score from a ratio between a number of terms in the historical transaction data that match the regular expression and that are PII and a total number of terms in the historical transaction data that match the regular expression, whether or not including PII.
8. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
storing a plurality of regular expressions for identifying personally identifiable information (PII), wherein each regular expression of the plurality of regular expressions comprises a sequence of symbols and characters expressing a string or pattern;
storing for each regular expression of the plurality of regular expressions a corresponding probability that the regular expression identifies the PII;
receiving content that includes a token;
comparing data within the token with each of the plurality of regular expressions;
determining, based on comparing each of the plurality of regular expressions with the data within the token, whether the data within the token matches a regular expression of the plurality of regular expressions;
based on determining that the data within the token matches the regular expression of the plurality of regular expressions, retrieving a probability associated the regular expression that matches the data within the token;
calculating a confidence score according to the probability associated the regular expression that matches the data within the token; and
masking the token in accordance with the confidence score.
9. The non-transitory computer-readable medium of claim 8 , wherein masking the token in accordance with the confidence score comprises:
comparing the confidence score to a threshold; and
based on determining that the confidence score is higher than the threshold, masking the token.
10. The non-transitory computer-readable medium of claim 8 , further comprising instructions for:
retrieving historical transaction data;
determining, from the historical transaction data, patterns for the PII; and
converting the patterns to the plurality of regular expressions.
11. The non-transitory computer-readable medium of claim 10 , further comprising instructions for generating clustered groups from the historical transaction data.
12. The non-transitory computer-readable medium of claim 8 wherein:
the content includes transaction data, the transaction data including one or more transaction records each corresponding to a respective transaction and each including a description of the transaction, and
the token is a part of the description and includes at least one of a number or a word.
13. The non-transitory computer-readable medium of claim 8 , wherein storing a plurality of regular expressions comprises storing the plurality of regular expressions in a database.
14. The non-transitory computer-readable medium of claim 10 , further comprising instructions for deriving the first confidence score from a ratio between a number of terms in the historical transaction data that match the regular expression and that are PII and a total number of terms in the historical transaction data that match the regular expression, whether or not including PII.
15. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
storing a plurality of regular expressions for identifying personally identifiable information (PII), wherein each regular expression of the plurality of regular expressions comprises a sequence of symbols and characters expressing a string or pattern;
storing for each regular expression of the plurality of regular expressions a corresponding probability that the regular expression identifies the PII;
receiving content that includes a token;
comparing data within the token with each of the plurality of regular expressions;
determining, based on comparing each of the plurality of regular expressions with the data within the token, whether the data within the token matches a regular expression of the plurality of regular expressions;
based on determining that the data within the token matches the regular expression of the plurality of regular expressions, retrieving a probability associated the regular expression that matches the data within the token;
calculating a confidence score according to the probability associated the regular expression that matches the data within the token; and
masking the token in accordance with the confidence score.
16. The system of claim 15 , wherein masking the token in accordance with the confidence score comprises:
comparing the confidence score to a threshold; and
based on determining that the confidence score is higher than the threshold, masking the token.
17. The system of claim 15 , further comprising instructions for:
retrieving historical transaction data;
determining, from the historical transaction data, patterns for the PII; and
converting the patterns to the plurality of regular expressions.
18. The system of claim 17 , further comprising instructions for generating clustered groups from the historical transaction data.
19. The system of claim 15 wherein:
the content includes transaction data, the transaction data including one or more transaction records each corresponding to a respective transaction and each including a description of the transaction, and
the token is a part of the description and includes at least one of a number or a word.
20. The system of claim 15 , wherein storing a plurality of regular expressions comprises storing the plurality of regular expressions in a database.