Automatic selection of blocking column for de-duplication
Blocking column selection can include determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set, and determining a second parameter for each column set. The second parameter can indicate block size for the column set. For each column set, a measure of blockability that is dependent upon at least the first parameter and the second parameter can be calculated using a processor. The plurality of column sets can be ranked according to the measures of blockability.
1. A system comprising:
a processor configured to initiate executable operations comprising:
determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set;
determining a second parameter for each column set, wherein the second parameter indicates block size for the column set;
for each column set, calculating a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set; and
ranking the plurality of column sets according to the measures of blockability.
2. The system of claim 1 , wherein the processor is further configured to initiate an executable operation comprising:
determining a third parameter for each column set, wherein the third parameter indicates an amount of null values in the column set, and wherein the third parameter is used in calculating the measure of blockability for the column set.
3. The system of claim 2 , wherein, for each column set, calculating a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set comprises:
determining a normalized version of each of the first parameter, the second parameter, and the third parameter; and
calculating a sum of the normalized first parameter and the normalized second parameter and dividing the sum by the normalized third parameter.
4. The system of claim 2 , wherein each column set comprises at least two columns, and wherein the processor is further configured to initiate an executable operation comprising:
determining a fourth parameter for each column set, wherein the fourth parameter indicates dependence between columns of the column set, and wherein the fourth parameter is used in calculating the measure of blockability for the column set.
5. The system of claim 4 , wherein, for each column set, calculating a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set comprises:
determining a normalized version of each of the first parameter, the second parameter, and the third parameter; and
calculating a first sum of the normalized first parameter and the normalized second parameter, calculating a second sum of the normalized third and the fourth parameter, and dividing the first sum by the second sum.
6. A computer program product for blocking column selection, the computer program product comprising:
a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code configured to determine a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set;
computer readable program code configured to determine a second parameter for each column set, wherein the second parameter indicates block size for the column set;
computer readable program code configured to, for each column set, calculate a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set; and
computer readable program code configured to rank the plurality of column sets according to the measures of blockability.
7. The computer program product of claim 6 , wherein the first parameter is a measure of entropy.
8. The computer program product of claim 6 , wherein the second parameter depends upon an average block size.
9. The computer program product of claim 6 , further comprising:
computer readable program code configured to determine a third parameter for each column set, wherein the third parameter indicates an amount of null values in the column set, and wherein the computer readable program code configured to calculate the measure of blockability for the column set uses the third parameter.
10. The computer program product of claim 9 , wherein, the computer readable program code configured to, for each column set, calculate a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set comprises:
computer readable program code configured to determine a normalized version of each of the first parameter, the second parameter, and the third parameter; and
computer readable program code configured to calculate a sum of the normalized first parameter and the normalized second parameter and dividing the sum by the normalized third parameter.
11. The computer program product of claim 6 , wherein each column set comprises at least two columns, the computer readable storage medium further comprising:
computer readable program code configured to determine a third parameter for each column set, wherein the third parameter indicates dependence between columns of the column set, and wherein the computer readable program code configured to calculate the measure of blockability for the column set uses the third parameter.
12. The computer program product of claim 9 , wherein each column set comprises at least two columns, the computer readable storage medium further comprising:
computer readable program code configured to determine a fourth parameter for each column set, wherein the fourth parameter indicates dependence between columns of the column set, and wherein the computer readable program code configured to calculate the measure of blockability for the column set uses the fourth parameter.
13. The computer program product of claim 12 wherein, the computer readable program code configured to, for each column set, calculate a measure of blockability that is dependent upon at least the first parameter and the second parameter for the column set comprises:
computer readable program code configured to determine a normalized version of each of the first parameter, the second parameter, and the third parameter; and
computer readable program code configured to calculate a first sum of the normalized first parameter and the normalized second parameter, calculate a second sum of the normalized third parameter and the fourth parameter, and divide the first sum by the second sum.