Automatic selection of blocking column for de-duplication
A method of blocking column selection can include determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set, and determining a second parameter for each column set. The second parameter can indicate block size for the column set. For each column set, a measure of blockability that is dependent upon at least the first parameter and the second parameter can be calculated using a processor. The plurality of column sets can be ranked according to the measures of blockability.
1. A method of blocking column selection, the method comprising:
determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set;
determining a second parameter for each column set, wherein the second parameter depends upon block size for the column set;
for each column set, calculating a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set using a processor; and
ranking each column set according to the measures of blockability.
2. The method of claim 1 , wherein the first parameter is a measure of entropy.
3. The method of claim 1 , wherein the second parameter depends upon an average block size.
4. The method of claim 3 , wherein the second parameter is an inverse log of the average block size.
5. The method of claim 1 , further comprising:
determining a third parameter for each column set, wherein the third parameter indicates an amount of null values in the column set, and wherein the third parameter is used in calculating the measure of blockability for the column set.
6. The method of claim 5 , wherein, for each column set, calculating a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set using a processor comprises:
determining a normalized version of each of the first parameter, the second parameter, and the third parameter; and
calculating a sum of the normalized first parameter and the normalized second parameter and dividing the sum by the normalized third parameter.
7. The method of claim 1 , wherein each column set comprises at least two columns, the method further comprising:
determining a third parameter for each column set, wherein the third parameter indicates dependence between columns of the column set, and wherein the third parameter is used in calculating the measure of blockability for the column set.
8. The method of claim 5 , wherein each column set comprises at least two columns, the method further comprising:
determining a fourth parameter for each column set, wherein the fourth parameter indicates dependence between columns of the column set, and wherein the fourth parameter is used in calculating the measure of blockability for the column set.
9. The method of claim 8 , wherein, for each column set, calculating a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set using a processor comprises:
determining a normalized version of each of the first parameter, the second parameter, and the third parameter; and
calculating a first sum of the normalized first parameter and the normalized second parameters, calculating a second sum of the normalized third parameter and the fourth parameter, and dividing the first sum by the second sum.
10. A method of blocking column selection, the method comprising:
determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set;
determining a second parameter for each column set, wherein the second parameter depends upon block size for the column set;
determining a third parameter for each column set, wherein the third parameter indicates an amount of null values in the column set;
determining a fourth parameter for each column set, wherein the fourth parameter indicates dependence between columns of the column set; and
for each column set, calculating a measure of blockability that depends upon the first parameter, the second parameter, the third parameter, and the fourth parameter.
11. The method of claim 10 , further comprising:
ranking each column set according to the measures of blockability.
12. The method of claim 10 , wherein, for each column set, calculating a measure of blockability that depends upon the first parameter, the second parameter, the third parameter, and the fourth parameter further comprises:
determining a normalized version of each of the first parameter, the second parameter, and the third parameter; and
calculating a first sum of the normalized first parameter and the normalized second parameter, calculating a second sum of the normalized third parameter and the fourth parameter, and dividing the first sum by the second sum.