Adaptive column set composition
A column set server adapts to data use patterns by data consumers by modifying how a table produced by a data producer is partitioned into separate column sets to reduce the waste incurred in accessing the data by multiple consumers of the data. For example, the column set server adjusts a column set distribution when a new consumer process is added, when one is retired, or when relative data set size ratios change.
1. A method comprising:
receiving notifications of a plurality of accesses by a plurality of data consumers to a table produced by a data producer, the notifications including columns of data of the table used by each of the data consumers, wherein at least a first and a second data consumer of the plurality of data consumers use at least a first column of the table, and wherein the first data consumer uses at least a second column of the table not used by the second data consumer;
analyzing by a column set server the plurality of accesses to determine an updated column set composition recommendation for the table, the analyzing comprising:
determining a total amount of waste for each of a plurality of combinations of column set partitions, each combination comprising a plurality of column set partitions having at least one column set partition having two or more columns of the table; and
selecting, from the plurality of combinations, the combination of column set partitions with the lowest total amount of waste to be the updated column set composition recommendation;
communicating the updated column set composition recommendation to the data producer for use in building a next iteration of the table; and
responsive to a change in the notifications of a plurality of accesses to the table, repeating the analyzing and the communicating steps of the method to adapt the column set composition of the table.
2. The method of claim 1 , wherein the analyzing comprises analyzing only combinations of column set partitions wherein the table is partitioned into five or fewer partitions.
3. The method of claim 1 , wherein the updated column set composition recommendation comprises identification of into how many sets the columns of the table should be divided and which columns of the table should be in each set.
4. The method of claim 1 , further comprising:
receiving notifications of a plurality of accesses to a plurality of other tables, and wherein the notifications of the accesses to the table and the notifications of accesses to the plurality of other tables each include a table identifier for the respective table.
5. The method of claim 1 , wherein a plurality of data producers produce a plurality of tables, each data producer associated with a respective producer identifier, and wherein the table is identified in the notifications by the table identifier and the producer identifier associated with the data producer that produced the table.
6. A non-transitory computer readable storage medium storing processor-executable instructions, the instructions configured to be executed by the processor to implement a method comprising:
receiving notifications of a plurality of accesses by a plurality of data consumers to a table produced by a data producer, the notifications including columns of data of the table used by each of the data consumers, wherein at least a first and a second data consumer of the plurality of data consumers use at least a first column of the table, and wherein the first data consumer uses at least a second column of the table not used by the second data consumer;
analyzing by a column set server the plurality of accesses to determine an updated column set composition recommendation for the table, the analyzing comprising:
determining a total amount of waste for each of a plurality of combinations of column set partitions, each combination comprising a plurality of column set partitions having at least one column set partition having two or more columns of the table; and
selecting, from the plurality of combinations, the combination of column set partitions with the lowest total amount of waste to be the updated column set composition recommendation;
communicating the updated column set composition recommendation to the data producer for use in building a next iteration of the table; and
responsive to a change in the notifications of a plurality of accesses to the table, repeating the analyzing and the communicating steps of the method to adapt the column set composition of the table.
7. The medium of claim 6 , wherein the analyzing comprises analyzing only combinations of column set partitions wherein the table is partitioned into five or fewer partitions.
8. The medium of claim 6 , wherein the updated column set composition recommendation comprises identification of into how many sets the columns of the table should be divided and which columns of the table should be in each set.
9. The medium of claim 6 , wherein the instructions are configured to be executed by the processor to implement a method further comprising:
receiving notifications of a plurality of accesses to a plurality of other tables, and wherein the notifications of the accesses to the table and the notifications of accesses to the plurality of other tables each include a table identifier for the respective table.
10. The medium of claim 6 , wherein a plurality of data producers produce a plurality of tables, each data producer associated with a respective producer identifier, and wherein the table is identified in the notifications by the table identifier and the producer identifier associated with the data producer that produced the table.
11. A system comprising:
a computer processor;
a computer readable storage medium storing processor-executable computer program instructions, the computer program instructions comprising instructions for:
receiving notifications of a plurality of accesses by a plurality of data consumers to a table produced by a data producer, the notifications including columns of data of the table used by each of the data consumers, wherein at least a first and a second data consumer of the plurality of data consumers use at least a first column of the table, and wherein the first data consumer uses at least a second column of the table not used by the second data consumer;
analyzing by a column set server the plurality of accesses to determine an updated column set composition recommendation for the table, the analyzing comprising:
determining a total amount of waste for each of a plurality of combinations of column set partitions, each combination comprising a plurality of column set partitions having at least one column set partition having two or more columns of the table; and
selecting, from the plurality of combinations, the combination of column set partitions with the lowest total amount of waste to be the updated column set composition recommendation;
communicating the updated column set composition recommendation to the data producer for use in building a next iteration of the table; and
responsive to a change in the notifications of a plurality of accesses to the table, repeating the analyzing and the communicating steps of the method to adapt the column set composition of the table.
12. The system of claim 11 , wherein the analyzing comprises analyzing only combinations of column set partitions wherein the table is partitioned into five or fewer partitions.
13. The system of claim 11 , wherein the updated column set composition recommendation comprises identification of into how many sets the columns of the table should be divided and which columns of the table should be in each set.
14. The system of claim 11 , wherein the instructions further comprise instructions for:
receiving notifications of a plurality of accesses to a plurality of other tables, and wherein the notifications of the accesses to the table and the notifications of accesses to the plurality of other tables each include a table identifier for the respective table.
15. The system of claim 11 , wherein a plurality of data producers produce a plurality of tables, each data producer associated with a respective producer identifier, and wherein the table is identified in the notifications by the table identifier and the producer identifier associated with the data producer that produced the table.