User data deidentification system for email addresses
A data deidentification system that extracts insights from user data and retains both the insights and user data in a form that complies with applicable data privacy and related standards. The system receives user data, which can include personal identifying information and other sensitive data governed by one or more standards, including standards specifying how the data can be used and how long it can be retained. From the data, the system extracts insights characterizing various aspects of the associated users. The system also selectively hashes portions of the data, obscuring the identity of associated users. Neither the insights nor the selectively hashed data identify individual users, and therefore they are not subject to the same standards and can be retained indefinitely. Later, after the standards-protected data has been discarded, the system can provide insight information in response to a request.
1 . A computer-implemented method of deidentifying an email address associated with a user, the method comprising:
receiving the email address associated with the user;
validating that the email address is in a standardized format;
extracting, from the email address, username data and domain data;
generating, based on the email address, insight data, wherein the insight data comprises a score indicating a likelihood that the email address was automatically generated;
identifying a protected portion of the username data and a protected portion of the domain data;
generating a hashed email address, a hashed username data, and a hashed domain data by applying a hash function to the email address, the protected portion of the username data, and the protected portion of the domain data, respectively;
storing, as deidentified email data, the insight data, the hashed email address, the hashed username data, and the hashed domain data;
receiving a request for user data, the request comprising a requested email address; and
providing deidentified user data based on the requested email address.
2 . The computer-implemented method of claim 1 , the method further comprising:
maintaining a plurality of deidentified user data, each associated with a user;
wherein providing deidentified user data based on the requested email address comprises:
generating a hashed requested email address by applying the hash function to the requested email address; and
identifying a deidentified user data, from the plurality of deidentified user data, based on the hashed requested email address, wherein the deidentified user data comprises the deidentified email data.
3 . The computer-implemented method of claim 1 , the method further comprising:
determining a timestamp associated with receiving the email address;
storing, in association with the deidentified email data, the email address and the timestamp;
evaluating, based on the timestamp and a retention time, whether a threshold time has elapsed; and
discarding, based on the evaluation, the email address, wherein the deidentified email data is retained beyond the threshold time.
4 . The computer-implemented method of claim 3 , the method further comprising:
determining a data privacy standard applicable to the email address, wherein the retention time is based on the data privacy standard.
5 . The computer-implemented method of claim 1 , wherein generating the insight data comprises:
generating a first score indicating a first likelihood that the domain data comprises automatically generated characters; and
generating a second score indicating a second likelihood that the username data comprises automatically generated characters.
6 . The computer-implemented method of claim 1 , wherein the username data comprises at least one non-protected characteristic characterizing a username of the email address, and wherein the at least one non-protected characteristic comprises a length of the username, a number of unique characters in the username, a proportion of vowels in the username, or an amount of repeated substrings in the username, and wherein the deidentified email data comprises the at least one non-protected characteristic in an unhashed format.
7 . The computer-implemented method of claim 1 , wherein the protected portion of the username data comprises at least one of a username, a set of letters in the username, or a set of numbers in the username.
8 . The computer-implemented method of claim 1 , the method further comprising:
receiving enrichment data associated with an email domain;
identifying a second deidentified email data that corresponds to the email domain; and
updating the second deidentified email data based on the enrichment data.
9 . The computer-implemented method of claim 1 , the method further comprising:
identifying, based on the hashed email address, deidentified user data; and
associating the deidentified email data with the deidentified user data, wherein the deidentified user data comprises insights associated with non-email address identifiers.
10 . The computer-implemented method of claim 1 , wherein the domain data comprises a non-protected portion, and wherein the stored deidentified email data further comprises the non-protected portion of the domain data.
11 . A non-transitory computer-readable medium carrying instructions that, when executed by a computing system, cause the computing system to perform operations for deidentifying data associated with a user, the operations comprising:
receiving an email address associated with the user;
validating that the email address is in a standardized format;
extracting, from the email address, username data and domain data;
generating, based on the email address, insight data, wherein the insight data comprises a score indicating a likelihood that the email address was automatically generated;
identifying a protected portion of the username data and a protected portion of the domain data;
generating a hashed email address, a hashed username data, and a hashed domain data by applying a hash function to the email address, the protected portion of the username data, and the protected portion of the domain data, respectively;
storing, as deidentified email data, the insight data, the hashed email address, the hashed username data, and the hashed domain data;
receiving a request for user data, the request comprising a requested email address; and
providing deidentified user data based on the requested email address.
12 . The non-transitory computer-readable medium of claim 11 , the operations further comprising:
maintaining a plurality of deidentified user data, each associated with a user;
wherein providing deidentified user data based on the requested email address comprises:
generating a hashed requested email address by applying the hash function to the requested email address; and
identifying a deidentified user data, from the plurality of deidentified user data, based on the hashed requested email address, wherein the deidentified user data comprises the deidentified email data.
13 . The non-transitory computer-readable medium of claim 11 , the operations further comprising:
determining a timestamp associated with receiving the email address;
storing, in association with the deidentified email data, the email address and the timestamp;
evaluating, based on the timestamp and a retention time, whether a threshold time has elapsed; and
discarding, based on the evaluation, the email address, wherein the deidentified email data is retained beyond the threshold time.
14 . The non-transitory computer-readable medium of claim 13 , the operations further comprising:
determining a data privacy standard applicable to the email address, wherein the retention time is based on the data privacy standard.
15 . The non-transitory computer-readable medium of claim 11 , wherein generating the insight data comprises:
generating a first score indicating a first likelihood that the domain data comprises automatically generated characters; and
generating a second score indicating a second likelihood that the username data comprises automatically generated characters.
16 . The non-transitory computer-readable medium of claim 11 , wherein the username data comprises at least one non-protected characteristic characterizing a username of the email address, and wherein the at least one non-protected characteristic comprises a length of the username, a number of unique characters in the username, a proportion of vowels in the username, or an amount of repeated substrings in the username, and wherein the deidentified email data comprises the at least one non-protected characteristic in an unhashed format.
17 . The non-transitory computer-readable medium of claim 11 , wherein the protected portion of the username data comprises at least one of a username, a set of letters in the username, or a set of numbers in the username.
18 . The non-transitory computer-readable medium of claim 11 , the operations further comprising:
receiving enrichment data associated with an email domain;
identifying a second deidentified email data that corresponds to the email domain; and
updating the second deidentified email data based on the enrichment data.
19 . The non-transitory computer-readable medium of claim 11 , the operations further comprising:
identifying, based on the hashed email address, deidentified user data; and
associating the deidentified email data with the deidentified user data, wherein the deidentified user data comprises insights associated with non-email address identifiers.
20 . The non-transitory computer-readable medium of claim 11 , wherein the domain data comprises a non-protected portion, and wherein the stored deidentified email data further comprises the non-protected portion of the domain data.