Adaptive column set composition
A column set server adapts to data use patterns by data consumers by modifying how a table produced by a data producer is partitioned into separate column sets to reduce the waste incurred in accessing the data by multiple consumers of the data. For example, the column set server adjusts a column set distribution when a new consumer process is added, when one is retired, or when relative data set size ratios change.
1. A computer-implemented method of adapting a column set composition of a table produced by a data producer and used by a plurality of data consumers, the method comprising:
receiving notifications of a plurality of accesses to the table by the plurality of data consumers, the notifications including the columns of data of the table used by each of the data consumers, wherein at least two of the plurality of data consumers use overlapping but different subsets of columns of the table, and wherein the table is identified in the notifications by a table identifier;
analyzing by a column set server the plurality of accesses to determine an updated column set composition recommendation for the table, the analyzing comprising:
analyzing a plurality of combinations of column set partitions;
determining a total amount of waste for each of the plurality of combinations of column set partitions by determining, for a respective combination of column set partitions, an amount of data read from the table but not used by the respective data consumer for each access of the plurality of accesses to the table by the plurality of data consumers and performance losses due to a respective data consumer accessing multiple files;
selecting the combination of column set partitions having the lowest total amount of waste to be the updated column set composition recommendation;
communicating the updated column set composition recommendation for the table to the data producer for use in building the table for a next iteration; and
responsive to a change in the notifications of a plurality of accesses to the table, repeating the receiving, the analyzing, and the communicating steps above to adapt the column set composition to the changed accesses to the table.
2. The method of claim 1 , wherein analyzing the plurality of combinations of column set partitions comprises analyzing only combinations of column set partitions wherein the table is partitioned into five or fewer partitions.
3. The method of claim 1 , wherein the updated column set composition recommendation comprises identification of into how many sets the columns should be divided and which columns should be in each set.
4. The method of claim 1 , further comprising:
receiving notifications of a plurality of accesses to a plurality of other tables, and wherein the notifications of the accesses to the table and the notifications of accesses to the plurality of other tables each include a table identifier for the respective table.
5. The method of claim 1 , wherein a plurality of data producers produce a plurality of tables, each data producer associated with a respective producer identifier, and wherein the table is identified in the notifications by the table identifier and the producer identifier associated with the data producer that produced the table.
6. A non-transitory computer readable storage medium executing computer program instructions for adapting a column set composition of a table produced by a data producer and used by a plurality of data consumers, the computer program instructions comprising instructions for:
receiving notifications of a plurality of accesses to the table by the plurality of data consumers, the notifications including the columns of data of the table used by each of the data consumers, wherein at least two of the plurality of data consumers use overlapping but different subsets of columns of the table, and wherein the table is identified in the notifications by a table identifier;
analyzing the plurality of accesses to determine an updated column set composition recommendation for the table, the analyzing comprising:
analyzing a plurality of combinations of column set partitions;
determining a total amount of waste for each of the plurality of combinations of column set partitions by determining, for a respective combination of column set partitions, an amount of data read from the table but not used by the respective data consumer for each access of the plurality of accesses to the table by the plurality of data consumers and performance losses due to a respective data consumer accessing multiple files;
selecting the combination of column set partitions having the lowest total amount of waste to be the updated column set composition recommendation;
communicating the updated column set composition recommendation for the table to the data producer for use in building the table for a next iteration; and
responsive to a change in the notifications of a plurality of accesses to the table, repeating the receiving, the analyzing, and the communicating steps above to adapt the column set composition to the changed accesses to the table.
7. The storage medium of claim 6 , wherein the instructions for analyzing the plurality of combinations of column set partitions comprise instructions for analyzing only combinations of column set partitions wherein the table is partitioned into five or fewer partitions.
8. The storage medium of claim 6 , wherein the updated column set composition recommendation comprises identification of into how many sets the columns should be divided and which columns should be in each set.
9. The storage medium of claim 6 , further comprising instructions for:
receiving notifications of a plurality of accesses to a plurality of other tables, and wherein the notifications of the accesses to the table and the notifications of accesses to the plurality of other tables each include a table identifier for the respective table.
10. The storage medium of claim 6 , wherein a plurality of data producers produce a plurality of tables, each data producer associated with a respective producer identifier, and wherein the table is identified in the notifications by the table identifier and the producer identifier associated with the data producer that produced the table.
11. A system comprising:
a computer processor;
a non-transitory computer readable storage medium storing processor-executable computer program instructions for adapting a column set composition of a table produced by a data producer and used by a plurality of data consumers, the instructions comprising instructions for:
receiving notifications of a plurality of accesses to the table by the plurality of data consumers, the notifications including the columns of data of the table used by each of the data consumers, wherein at least two of the plurality of data consumers use overlapping but different subsets of columns of the table, and wherein the table is identified in the notifications by a table identifier;
analyzing the plurality of accesses to determine an updated column set composition recommendation for the table, the analyzing comprising:
analyzing a plurality of combinations of column set partitions;
determining a total amount of waste for each of the plurality of combinations of column set partitions by determining, for a respective combination of column set partitions, an amount of data read from the table but not used by the respective data consumer for each access of the plurality of accesses to the table by the plurality of data consumers and performance losses due to a respective data consumer accessing multiple files;
selecting the combination of column set partitions having the lowest total amount of waste to be the updated column set composition recommendation;
communicating the updated column set composition recommendation for the table to the data producer for use in building the table for a next iteration; and
responsive to a change in the notifications of a plurality of accesses to the table, repeating the receiving, the analyzing, and the communicating steps above to adapt the column set composition to the changed accesses to the table.
12. The system of claim 11 , wherein the instructions for analyzing the plurality of combinations of column set partitions comprise instructions for analyzing only combinations of column set partitions wherein the table is partitioned into five or fewer partitions.
13. The system of claim 11 , wherein the updated column set composition recommendation comprises identification of into how many sets the columns should be divided and which columns should be in each set.
14. The system of claim 11 , wherein the instructions further comprise instructions for:
receiving notifications of a plurality of accesses to a plurality of other tables, and wherein the notifications of the accesses to the table and the notifications of accesses to the plurality of other tables each include a table identifier for the respective table.
15. The system of claim 11 , wherein a plurality of data producers produce a plurality of tables, each data producer associated with a respective producer identifier, and wherein the table is identified in the notifications by the table identifier and the producer identifier associated with the data producer that produced the table.