Database key identification

1. A method including:

receiving by a data processing system a first data set that includes a plurality of records;

identifying a set of one or more first fields as a potential key field representing a potential primary key of the first data set;

for a value in at least one of the one or more first fields,

performing one or more hash functions on the value to generate at least one hash value;

identifying a set of bits for a filter key; and

based on the at least one hash value, setting at least one value for one or more of the bits for the filter key;

based on set values of the bits, generating a filter key for the value in at least one of the one or more first fields;

identifying one or more filter keys that represent one or more values in the set of the one or more first fields representing the potential primary key;

generating a filter mask from the one or more filter keys representing the potential primary key field, wherein a dataset passes the filter mask, when the passed dataset has a potential primary key—foreign key relationship to the first data set;

storing, in memory, the filter mask that stores a set of bits that are generated from the one or more filter keys that represent the one or more values in the set of one or more first fields representing the potential primary key;

receiving a second data set that includes a plurality of records that have one or more second fields representing a potential foreign key;

wherein when the second data set passes the filter mask, the data processing system indicates a potential primary key—foreign key relationship between the potential key field of the first data set and the one more second fields in the second data set;

filtering by the data processing system bits representing values in the one or more second fields in the records in the second data set through the filter mask generated from the one or more filter keys that represent the one or more values in the set of one or more first fields identified as the potential key field;

determining whether the values in the one or more second fields in the records in the second data set have corresponding one or more values that pass the filter mask by being represented in the filter mask; and

when one or more values in the one or more second fields are included in the set of values represented in the filter mask,

indicating a potential primary key—foreign key relationship between the potential key field of the first data set and the one more second fields in the second data set.

2. The method of claim 1 , further including

determining a count of a number of records each having a value associated with the one or more second fields in the records in the second data set that passes the filter mask;

storing the count in a profile; and

determining the Sørensen-Dice coefficient of the set of values in the filter mask and the records in the second data set having a value associated with the one or more second fields.

3. The method of claim 1 , further includes:

producing for a given record a filter key that is based on values in the one or more first fields of the given record in the first data set; and

generating the filter mask based on filter keys produced for the records in the first data set by combining the filter keys according to a Boolean operation.

4. The method of claim 3 , wherein generating a filter key for a corresponding value includes:

generating a hash value for the corresponding value;

segmenting the hash value into a predetermined number of integers; and

generating the filter key by setting bits in a bit vector based on the integers.

5. The method of claim 3 , wherein generating the filter mask further includes performing a binary operation on each of a plurality of generated filter keys.

6. The method of claim 3 , further including:

determining whether the values in the one or more second fields in the records in the second data set have corresponding one or more values that pass the filter mask, including:

generating one or more second filter keys for the one or more values associated with the one or more second fields in the second data set; and

comparing the one or more second filter keys to the filter mask.

7. A non-transitory computer storage medium encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a first data set that includes a plurality of records;

identifying a set of one or more first fields as a potential key field representing a potential primary key of the first data set;

for a value in at least one of the one or more first fields,

performing one or more hash functions on the value to generate at least one hash value;

identifying a set of bits for a filter key; and

based on the at least one hash value, setting at least one value for one or more of the bits for the filter key;

based on set values of the bits, generating a filter key for the value in at least one of the one or more first fields;

identifying one or more filter keys that represent one or more values in the set of the one or more first fields representing the potential primary key;

generating a filter mask from the one or more filter keys representing the potential primary key field, wherein a dataset passes the filter mask, when the passed dataset has a potential primary key—foreign key relationship to the first data set;

storing, in memory, the filter mask that stores a set of bits that are generated from the one or more filter keys that represent the one or more values in the set of one or more first fields representing the potential primary key;

receiving a second data set that includes a plurality of records that have one or more second fields representing a potential foreign key;

wherein when the second data set passes the filter mask, the data processing system indicates a potential primary key—foreign key relationship between the potential key field of the first data set and the one more second fields in the second data set;

filtering bits representing values in the one or more second fields in the records in the second data set through the filter mask generated from the one or more filter keys that represent the one or more values in the set of one or more first fields identified as the potential key field;

determining whether the values in the one or more second fields in the records in the second data set have corresponding one or more values that pass the filter mask by being represented in the filter mask; and

when one or more values in the one or more second fields are included in the set of values represented in the filter mask,

indicating a potential primary key—foreign key relationship between the potential key field of the first data set and the one more second fields in the second data set.

8. The medium of claim 7 , further including

determining a count of a number of records each having a value associated with the one or more second fields in the records in the second data set that passes the filter mask;

storing the count in a profile; and

determining the Sørensen-Dice coefficient of the set of values in the filter mask and the records in the second data set having a value associated with the one or more second fields.

9. The medium of claim 7 , wherein the operations further include:

producing for a given record a filter key that is based on values in the one or more first fields of the given record in the first data set; and

generating the filter mask based on filter keys produced for the records in the first data set by combining the filter keys according to a Boolean operation.

10. The medium of claim 9 , wherein generating a filter key for a corresponding value includes:

generating a hash value for the corresponding value;

segmenting the hash value into a predetermined number of integers; and

generating the filter key by setting bits in a bit vector based on the integers.

11. The medium of claim 9 , wherein generating the filter mask further includes performing a binary operation on each of a plurality of generated filter keys.

12. The medium of claim 9 , wherein the operations further include:

determining whether the values in the one or more second fields in the records in the second data set have corresponding one or more values that pass the filter mask, including:

generating one or more second filter keys for the one or more values associated with the one or more second fields in the second data set; and

comparing the one or more second filter keys to the filter mask.

13. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a first data set that includes a plurality of records;

identifying a set of one or more first fields as a potential key field representing a potential primary key of the first data set;

for a value in at least one of the one or more first fields,

performing one or more hash functions on the value to generate at least one hash value;

identifying a set of bits for a filter key; and

based on the at least one hash value, setting at least one value for one or more of the bits for the filter key;

based on set values of the bits, generating a filter key for the value in at least one of the one or more first fields;

identifying one or more filter keys that represent one or more values in the set of the one or more first fields representing the potential primary key;

generating a filter mask from the one or more filter keys representing the potential primary key field, wherein a dataset passes the filter mask, when the passed dataset has a potential primary key—foreign key relationship to the first data set;

storing, in memory, the filter mask that stores a set of bits that are generated from the one or more filter keys that represent the one or more values in the set of one or more first fields representing the potential primary key;

receiving a second data set that includes a plurality of records that have one or more second fields representing a potential foreign key;

wherein when the second data set passes the filter mask, the data processing system indicates a potential primary key—foreign key relationship between the potential key field of the first data set and the one more second fields in the second data set

filtering bits representing values in the one or more second fields in the records in the second data set through the filter mask generated from the one or more filter keys that represent the one or more values in the set of one or more first fields identified as the potential key field;

determining whether the values in the one or more second fields in the records in the second data set have corresponding one or more values that pass the filter mask by being represented in the filter mask; and

when one or more values in the one or more second fields are included in the set of values represented in the filter mask,

indicating a potential primary key—foreign key relationship between the potential key field of the first data set and the one more second fields in the second data set.

14. The system of claim 13 , further including

determining a count of a number of records each having a value associated with the one or more second fields in the records in the second data set that passes the filter mask;

storing the count in a profile; and

determining the Sørensen-Dice coefficient of the set of values in the filter mask and the records in the second data set having a value associated with the one or more second fields.

15. The system of claim 13 , wherein the operations further include:

producing for a given record a filter key that is based on values in the one or more first fields of the given record in the first data set; and

generating the filter mask based on filter keys produced for the records in the first data set by combining the filter keys according to a Boolean operation.

16. The system of claim 15 , wherein generating a filter key for a corresponding value includes:

generating a hash value for the corresponding value;

segmenting the hash value into a predetermined number of integers; and

generating the filter key by setting bits in a bit vector based on the integers.

17. The system of claim 15 , wherein generating the filter mask further includes performing a binary operation on each of a plurality of generated filter keys.

18. The system of claim 15 , wherein the operations further include:

determining whether the values in the one or more second fields in the records in the second data set have corresponding one or more values that pass the filter mask including:

generating one or more second filter keys for the one or more values associated with the one or more second fields in the second data set; and

comparing the one or more second filter keys to the filter mask.