User data deidentification system for IP addresses
A data deidentification system that extracts insights from user data and retains both the insights and user data in a form that complies with applicable data privacy and related standards. The system receives user data, which can include personal identifying information and other sensitive data governed by one or more standards, including standards specifying how the data can be used and how long it can be retained. From the data, the system extracts insights characterizing various aspects of the associated users. The system also selectively hashes portions of the data, obscuring the identity of associated users. Neither the insights nor the selectively hashed data identify individual users, and therefore they are not subject to the same standards and can be retained indefinitely. Later, after the standards-protected data has been discarded, the system can provide insight information in response to a request.
1 . A computer-implemented method of deidentifying an IP address associated with a user, the method comprising:
receiving transaction data associated with the user,
wherein the transaction data comprises an internet protocol (IP) address;
determining whether the IP address is an IPv4 address or an IPv6 address;
generating a plurality of classless inter-domain routing (CIDR) blocks corresponding to the IP address,
wherein the number of the generated CIDR blocks is based on whether the IP address is an IPv4 address or an IPv6 address;
hashing the plurality of CIDR blocks to generate a plurality of hashed CIDR blocks;
hashing the IP address to generate a hashed IP address;
extracting insight data from the transaction data,
wherein the insight data comprises non-identifying data characterizing the transaction data;
storing, as deidentified transaction data, the insight data, the hashed IP address, and the plurality of hashed CIDR blocks;
receiving a request for transaction data, the request comprising a requested IP address; and
providing deidentified transaction data based on the requested IP address.
2 . The computer-implemented method of claim 1 , the method further comprising:
identifying, based on the hashed IP address, an existing user record; and
associating the deidentified transaction data with the existing user record.
3 . The computer-implemented method of claim 1 , wherein the IP address comprises a number of bits, and wherein determining whether the IP address is an IPv4 address or an IPv6 address is based on the number of bits in the IP address.
4 . The computer-implemented method of claim 1 , wherein 32 CIDR blocks are generated when it is determined that the IP address is an IPv4 address.
5 . The computer-implemented method of claim 1 , wherein 128 CIDR blocks are generated when it is determined that the IP address is an IPv6 address.
6 . The computer-implemented method of claim 1 , wherein the method further comprises:
identifying a number of bits in the IP address; and
performing error correction on the IP address when the number of bits is other than 32 bits or 128 bits.
7 . The computer-implemented method of claim 1 , wherein the method further comprises:
identifying a maximum segment in the IP address, wherein the maximum segment represents a segment with a numeric value greater than the numeric values of other segments in the IP address; and
flagging the IP address for manual review when the maximum segment is greater than 255 for an IPv4 address or greater than FFFFF in hexadecimal for an IPv6 address.
8 . The computer-implemented method of claim 1 , wherein the method further comprises validating the IP address based on:
determining when a first condition is met based on determining a threshold number of address segments, wherein the threshold number of address segments corresponding to an IPv4 address is 4, and wherein the threshold number of address segments corresponding to an IPv6 address is 8;
determining when a second condition is met based on determining allowable characters associated with the IP address, wherein the allowable characters for an IPv4 address comprise only digit characters between 0 and 9, and wherein the allowable characters for an IPv6 address comprise only digit characters between 0 and 9 and letter characters ‘a’, ‘b’, ‘c’, ‘d’, ‘e’, and ‘f’;
determining when a third condition is met based on determining a maximum segment value in the IP address, wherein the maximum segment value is greater than other segment values in the IP address, and based further on whether the maximum segment value exceeds 255 for an IPv4 address or exceeds FFFFF in hexadecimal for an IPv6 address; and
generating a validity status, associated with the IP address, wherein the validity status is either valid or invalid and is valid when the IP address meets the first condition, the second condition, and the third condition and is invalid otherwise.
9 . The computer-implemented method of claim 1 , wherein the method further comprises:
identifying port information associated with the IP address,
wherein the port information comprises an integer corresponding to a network protocol; and
removing, prior to generating the plurality of CIDR blocks, the port information associated with the IP address.
10 . The computer-implemented method of claim 1 , wherein the insight data further comprises an indication of whether the IP address belongs to a public or private space, and wherein the method further comprises generating the indication based on:
determining that the IP address falls within a reserved IP address range; and
including a reference to the reserved IP address range in the insight data,
wherein the reference includes the reserved IP address range.
11 . The computer-implemented method of claim 1 , further comprising:
identifying a timestamp corresponding to the transaction data,
wherein the timestamp indicates a time of receipt;
storing, in association with the deidentified transaction data, the IP address; and
discarding the IP address in response to determining that a threshold time period has passed subsequent to the time of receipt.
12 . The computer-implemented method of claim 11 , wherein the threshold time period is based on a retention time associated with a data privacy standard.
13 . The computer-implemented method of claim 1 , wherein hashing the plurality of CIDR blocks to generate the plurality of hashed CIDR blocks comprises applying a hashing algorithm to each CIDR block in the plurality of CIDR blocks.
14 . The computer-implemented method of claim 1 , wherein hashing the IP address to generate the hashed IP address comprises applying a hashing algorithm to the IP address.
15 . The computer-implemented method of claim 1 , wherein extracting insight data from the transaction data, further comprises:
removing personally identifiable information (PII) from the transaction data prior to extracting the insight data, based on:
identifying the PII, wherein the PII can be used to identify the user associated with the transaction data; and
removing or obfuscating the PII associated with the transaction data; and
extracting the insight data from the transaction data.