User data deidentification system
A data deidentification system that extracts insights from user data, and retains both the insights and user data in a form that complies with applicable data privacy and related standards. The system receives user data, which can include personal identifying information and other sensitive data governed by one or more standards, including standards specifying how the data can be used and how long it can be retained. From the data, the system extracts insights characterizing various aspects of the associated users. The system also selectively hashes portions of the data, obscuring the identity of associated users. Neither the insights nor the selectively hashed data identify individual users, and therefore they are not subject to the same standards and can be retained indefinitely. Later, after the standards-protected data has been discarded, the system can provide insight information in response to a request.
1. A computer-implemented method of deidentifying data associated with a user, the method comprising:
receiving the data associated with the user, the data comprising a protected portion and a non-protected portion,
wherein the protected portion comprises an identifier associated with the user that is subject to a data privacy standard;
pre-processing the received data, wherein pre-processing the received data comprises:
converting the identifier associated with the user to a standardized format,
cleansing at least some of the received data to remove extraneous content, and
timestamping the received data to indicate a time of receipt;
generating insight data based on the identifier;
generating a selectively hashed version of the identifier by:
determining a type associated with the identifier,
determining a hash function based on the determined type, and
applying the hash function to at least a portion of the identifier;
discarding the protected portion of the received data in response to determining that a threshold time period has passed subsequent to the time of receipt; and
retaining the insight data and the selectively hashed version of the identifier as deidentified data, wherein the deidentified data is not subject to the data privacy standard.
2. The computer-implemented method of claim 1 , wherein the protected portion of the received data comprises personal identifiable information (PII).
3. The computer-implemented method of claim 1 , wherein the identifier associated with the user is a phone number, a personal identifier, a device identifier, an email address, an internet protocol (IP) address, a mailing address, or a physical address.
4. The computer-implemented method of claim 1 :
wherein the identifier associated with the user comprises an email address,
wherein generating the insight data based on the identifier comprises generating a score indicating a likelihood that the email address was automatically generated, and
wherein generating the selectively hashed version of the identifier comprises hashing a username portion of the email address and not hashing a domain name portion of the email address.
5. The computer-implemented method of claim 1 :
wherein the identifier associated with the user comprises a physical address or a mailing address, and
wherein generating the insight data based on the identifier comprises generating geocoded data based on the physical address or the mailing address.
6. The computer-implemented method of claim 1 , wherein generating the insight data comprises receiving supplemental data from a third-party service.
7. The computer-implemented method of claim 1 , further comprising:
determining the data privacy standard applicable to the identifier; and
determining a retention time associated with the determined data privacy standard,
wherein the threshold time period is based on the retention time.
8. The computer-implemented method of claim 1 , wherein pre-processing the received data includes correcting an error in the received data.
9. The computer-implemented method of claim 1 , further comprising:
identifying, based on the selectively hashed version of the identifier, an existing user data in a data record; and
associating the deidentified data with the existing user data,
wherein the existing user data comprises different insights associated with different identifiers of the existing user.
10. The computer-implemented method of claim 1 , further comprising:
receiving a request for deidentified data associated with a user identifier;
generating a deidentified equivalent of the user identifier;
identifying retained deidentified data associated with the deidentified equivalent of the user identifier; and
providing the identified retained deidentified data.
11. A non-transitory computer-readable medium carrying instructions that, when executed by a computing system, cause the computing system to perform operations for deidentifying data associated with a user, the operations comprising:
receiving the data associated with the user, the data comprising a protected portion and a non-protected portion,
wherein the protected portion comprises an identifier associated with the user that is subject to a data privacy standard;
pre-processing the received data, wherein pre-processing the received data comprises:
converting the identifier associated with the user to a standardized format,
cleansing at least some of the received data to remove extraneous content, and
timestamping the received data to indicate a time of receipt;
generating insight data based on the identifier;
generating a selectively hashed version of the identifier by:
determining a type associated with the identifier,
determining a hash function based on the determined type, and
applying the hash function to at least a portion of the identifier;
discarding the protected portion of the received data in response to determining that a threshold time period has passed subsequent to the time of receipt; and
retaining the insight data and the selectively hashed version of the identifier as deidentified data, wherein the deidentified data is not subject to the data privacy standard.
12. The computer-readable medium of claim 11 , wherein the protected portion of the received data comprises personal identifiable information (PII).
13. The computer-readable medium of claim 11 , wherein the identifier associated with the user is a phone number, a personal identifier, a device identifier, an email address, an internet protocol (IP) address, a mailing address, or a physical address.
14. The computer-readable medium of claim 11 :
wherein the identifier associated with the user comprises an email address,
wherein generating the insight data based on the identifier comprises generating a score indicating a likelihood that the email address was automatically generated, and
wherein generating the selectively hashed version of the identifier comprises hashing a username portion of the email address and not hashing a domain name portion of the email address.
15. The computer-readable medium of claim 11 :
wherein the identifier associated with the user comprises a physical address or a mailing address, and
wherein generating the insight data based on the identifier comprises generating geocoded data based on the physical address or the mailing address.
16. The computer-readable medium of claim 11 , wherein generating the insight data comprises receiving supplemental data from a third-party service.
17. The computer-readable medium of claim 11 , further comprising:
determining the data privacy standard applicable to the identifier; and
determining a retention time associated with the determined data privacy standard,
wherein the threshold time period is based on the retention time.
18. The computer-readable medium of claim 11 , wherein pre-processing the received data includes correcting an error in the received data.
19. The computer-readable medium of claim 11 , wherein the operations further comprise:
identifying, based on the selectively hashed version of the identifier, an existing user data in a data record; and
associating the deidentified data with the existing user data,
wherein the existing user data comprises different insights associated with different identifiers of the existing user.
20. The computer-readable medium of claim 11 , wherein the operations further comprise:
receiving a request for deidentified data associated with a user identifier;
generating a deidentified equivalent of the user identifier;
identifying retained deidentified data associated with the deidentified equivalent of the user identifier; and
providing the identified retained deidentified data.