Secure multi-party reach and frequency estimation
Systems and methods for generating min-increment counting bloom filters to determine count and frequency of device identifiers and attributes in a networking environment are disclosed. The system can maintain a set of data records including device identifiers and attributes associated with device in a network. The system can generate a vector comprising coordinates corresponding to counter registers. The system can identify hash functions to update a counting bloom filter. The system can hash the data records to extract index values pointing to a set of counter registers. The system can increment the positions in the min-increment counting bloom filter corresponding to the minimum values of the counter registers. The system can obtain an aggregated public key comprising a public key. The system can encrypt the counter registers using the aggregated shared key to generate an encrypted vector. The system can transmit the encrypted vector to a networked worker computing device.
1. A method of generating an encrypted data structure representative of a set of identifiers having attributes that satisfy target criteria for secure and computationally efficient transmission, comprising:
maintaining, by a data processing system comprising one or more processors and a memory, in a database, a set of identifiers, each of the set of identifiers comprising an attribute;
generating, by the data processing system, a vector data structure comprising a plurality of coordinates each corresponding to a respective one of a plurality of counter registers;
accessing, by the data processing system, each of the plurality of counter registers that correspond to a plurality of register identifiers to identify a set of counter registers that satisfy a minimum value threshold, wherein identifying the set of counter registers that satisfy the minimum value threshold comprises:
determining a value for each respective counter register of the plurality of counter registers to determine a minimum value;
comparing the value for each respective counter register of the plurality of counter registers;
identifying one or more counter registers of the plurality of counter registers as satisfying the minimum value threshold if the value of the respective counter register is equal to the minimum value; and
identifying one or more counter registers of the plurality of counter registers as not satisfying the minimum value threshold if the value of the respective counter register is not equal to the minimum value;
incrementing, by the data processing system, each of the set of counter registers that satisfy the minimum value threshold;
not incrementing, by the data processing system, each of the set of counter registers that do not satisfy the minimum value threshold;
encrypting, by the data processing system, the vector data structure to create an encrypted data structure, such that the encrypted data structure can be combined with a second encrypted data structure; and
transmitting, by the data processing system, the encrypted data structure to a worker computing device.
2. The method of claim 1 , further comprising:
receiving, by the data processing system, a first identifier comprising a first device attribute; and
storing, by the data processing system, the first identifier comprising the first device attribute as a member of the set of identifiers.
3. The method of claim 1 , further comprising:
hashing, by the data processing system, a identifier of the set of identifiers using each of a plurality of hash functions to generate a plurality of hashed data record values;
extracting, by the data processing system, a plurality of register identifiers from the plurality of hashed data record values, each of the plurality of register identifiers corresponding to a respective counter register of the plurality of counter registers; and
identifying, by the data processing system, a uniformly distributed hash function as one of the plurality of hash functions, wherein the uniformly distributed hash function outputs uniformly distributed values.
4. The method of claim 3 , wherein hashing the identifier of the set of identifiers further comprises storing, by the data processing system, in the database, each of the plurality of hashed data record values in association with the identifier.
5. The method of claim 4 , wherein extracting the plurality of register identifiers further comprises performing a modulus operation on each of the plurality of hashed data record values using a number of the plurality of counter registers.
6. The method of claim 3 , wherein the vector data structure is first vector in a matrix data structure comprising the first vector and a second vector, and wherein generating the vector data structure further comprises:
selecting, by the data processing system, a selected vector using a hash function of the plurality of hash functions and the identifier of the set of identifiers, wherein the selected vector is one of the first vector or the second vector; and
updating, by the data processing system, a coordinate of the selected vector of the matrix data structure using the hash function and the identifier of the set of identifiers.
7. The method of claim 6 , wherein selecting the selected vector further comprises:
hashing, by the data processing system, the identifier of the set of identifiers to generate a hashed identifier;
determining, by the data processing system, a number of least significant bits of the hashed identifier that satisfy a predetermined bit value; and
selecting, by the data processing system, the first vector as the selected vector or the second vector as the selected vector, based on the number of least significant bits of the hashed identifier that satisfy the predetermined bit value.
8. The method of claim 7 , wherein updating the coordinate of the selected vector of the matrix data structure further comprises:
performing, by the data processing system, a modulus operation on the hashed identifier to calculate a counter register index value;
selecting, by the data processing system, the coordinate using the counter register index value; and
incrementing, by the data processing system, the coordinate selected using the counter register index value.
9. The method of claim 1 , comprising:
hashing, by the data processing system, a identifier of the set of identifiers using each of a plurality of hash functions to generate a plurality of hashed data record values; and
extracting, by the data processing system, a plurality of register identifiers from the plurality of hashed data record values, each of the plurality of register identifiers corresponding to a respective counter register of the plurality of counter registers.
10. A system for generating an encrypted data structure representative of a set of identifiers having attributes that satisfy target criteria for secure and computationally efficient transmission, comprising:
a data processing system comprising one or more processors and a memory, the data processing system configured to:
maintain, by a data processing system comprising one or more processors and a memory, in a database, a set of identifiers, each of the set of identifiers comprising an attribute;
generate, by the data processing system, a vector data structure comprising a plurality of coordinates each corresponding to a respective one of a plurality of counter registers;
access, by the data processing system, each of the plurality of counter registers that correspond to a plurality of register identifiers to identify a set of counter registers that satisfy a minimum value threshold, wherein identifying the set of counter registers that satisfy the minimum value threshold comprises:
determine a value for each respective counter register of the plurality of counter registers to determine a minimum value;
compare the value for each respective counter register of the plurality of counter registers;
identify one or more counter registers of the plurality of counter registers as satisfying the minimum value threshold if the value of the respective counter register is equal to the minimum value; and
identifying one or more counter registers of the plurality of counter registers as not satisfying the minimum value threshold if the value of the respective counter register is not equal to the minimum value;
increment, by the data processing system, each of the set of counter registers that satisfy the minimum value threshold;
not increment, by the data processing system, each of the set of counter registers that do not satisfy the minimum value threshold;
encrypt, by the data processing system, the vector data structure to create an encrypted data structure, such that the encrypted data structure can be combined with a second encrypted data structure; and
transmit, by the data processing system, the encrypted data structure to a worker computing device.
11. The system of claim 10 , wherein the data processing system is further configured to:
receive a first identifier comprising a first device attribute; and
store the first identifier comprising the first device attribute as a member of the set of identifiers.
12. The system of claim 10 , wherein the data processing system is further configured to:
hash, by the data processing system, a identifier of the set of identifiers using each of a plurality of hash functions to generate a plurality of hashed data record values;
extract, by the data processing system, a plurality of register identifiers from the plurality of hashed data record values, each of the plurality of register identifiers corresponding to a respective counter register of the plurality of counter registers; and
identify, by the data processing system, a uniformly distributed hash function as one of the plurality of hash functions, wherein the uniformly distributed hash function outputs uniformly distributed values.
13. The system of claim 12 , wherein the data processing system is further configured to store, in the database, each of the plurality of hashed data record values in association with the identifier.
14. The system of claim 12 , wherein the data processing system is further configured to perform a modulus operation on each of the plurality of hashed data record values using a number of the plurality of counter registers.
15. The system of claim 12 , wherein the vector data structure is a first vector in a matrix data structure comprising the first vector and a second vector, and wherein the data processing system is further configured to:
select a selected vector using a hash function of the plurality of hash functions and the identifier of the set of identifiers, wherein the selected vector is one of the first vector or the second vector; and
update a coordinate of the selected vector of the matrix data structure using the hash function and the identifier of the set of identifiers.
16. The system of claim 15 , wherein the data processing system is further configured to:
hash the identifier of the set of identifiers to generate a hashed identifier; determine a number of least significant bits of the hashed identifier that satisfy a predetermined bit value; and
select the first vector as the selected vector or the second vector as the selected vector, based on the number of least significant bits of the hashed identifier that satisfy the predetermined bit value.
17. The system of claim 16 , wherein the data processing system is further configured to:
perform a modulus operation on the hashed identifier to calculate a counter register index value;
select the coordinate using the counter register index value; and increment the coordinate selected using the counter register index value.
18. The system of claim 10 , wherein the data processing system is further configured to:
hash, by the data processing system, a identifier of the set of identifiers using each of a plurality of hash functions to generate a plurality of hashed data record values; and
extract, by the data processing system, a plurality of register identifiers from the plurality of hashed data record values, each of the plurality of register identifiers corresponding to a respective counter register of the plurality of counter registers.