Systems and methods for redistributing data in a relational database
Systems and methods for redistributing data in a relational database are disclosed. In one embodiment, the database includes a plurality of rows of data distributed across a plurality of slices of a table in the database. The database system is configured to distribute the rows of data across the slices according to a first function based on one or more columns of the table. The database system monitors at least one database statistic indicative of variation in a distribution of the rows of data across the slices and detects a redistribution condition based on the at least one monitored database statistic. The database system is further configured to respond to the detected redistribution condition by redistributing the rows of data across the slices according to a second function based on a different number of columns than the first function.
1. A method of redistributing data in a distributed database comprising a plurality of rows of data distributed across a plurality of slices of a table in the database, the method comprising:
distributing the rows of data across the slices unevenly according to a first hash function value of at least one column of each of the rows of the table;
monitoring at least one database statistic wherein the at least one database statistic comprises at least one of: a hash range occupancy ratio for each slice in said plurality of slices, a list of hot values for each column of each slice in said plurality of slices, a number of distinct values for each column of each slice in said plurality of slices, probabilistic data distribution of each column in each slice in said plurality of slices, or a quantile distribution of values for each column of each slice in said plurality of slices;
detecting a redistribution condition based on the at least one monitored database statistic; and
responding to the detected redistribution condition by redistributing the rows of data across the slices according to a second hash function wherein the second hash function is based on a different number of column values of the table than the first hash function;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , wherein the first hash function is based on a set of one or more columns, and the second hash function is based on the set of one or more columns of the first hash function, plus at least one additional column.
3. The method of claim 1 , wherein the first hash function is based on a set of one or more columns, and the second hash function is based on the set of one or more columns of the first hash function, minus at least one column.
4. The method of claim 1 , wherein said monitoring comprises periodically evaluating the at least one database statistic, wherein detecting the redistribution condition comprises detecting that the at least one database statistic exceeds a threshold value.
5. The method of claim 1 , wherein said distributing the rows comprises:
hashing one or more column values in a row into a column value hash;
determining one hash range, out of a plurality of hash ranges, into which the column value hash falls; and
assigning the row to one of the slices based on the determined hash range.
6. The method of claim 1 , wherein the at least one database statistic comprises, for each of the most common sets of one or more values for a set of one or more columns of the table, a total number of rows containing the set of one or more values for the set of one or more columns.
7. The method of claim 1 , wherein the at least one database statistic comprises a number of rows falling into each of a plurality of quantiles of potential values for a column.
8. The method of claim 1 , further comprising splitting one of the slices into a plurality of new slices, wherein detecting the redistribution condition further comprises detecting the splitting of the slice into the plurality of new slices.
9. The method of claim 1 , wherein the database slices continue to be available for database transactions while the data in the first slice is redistributed to the new slices.
10. A distributed database system comprising a plurality of rows of data distributed across a plurality of slices of a table in the database, and one or more processors configured to:
distribute the rows of data across the slices unevenly according to a first hash function value of at least one column of each of the rows of the table;
monitor at least one database statistic wherein the at least one database statistic comprises at least one of: a hash range occupancy ratio for each slice in said plurality of slices, a list of hot values for each column of each slice in said plurality of slices, a number of distinct values for each column of each slice in said plurality of slices, probabilistic data distribution of each column in each slice in said plurality of slices, or a quantile distribution of values for each column of each slice in said plurality of slices;
detect a redistribution condition based on the at least one monitored database statistic; and
respond to the detected redistribution condition by redistributing the rows of data across the slices according to a second hash function wherein the second hash function is based on a different number of column values of the table than the first hash function.
11. The distributed database system of claim 10 , wherein the first hash function is based on a set of one or more columns, and the second hash function is based on the set of one or more columns of the first hash function, plus at least one additional column.
12. The distributed database system of claim 10 , wherein the first hash function is based on a set of one or more columns, and the second hash function is based on the set of one or more columns of the first hash function, minus at least one column.
13. The distributed database system of claim 10 , wherein the one or more processors are configured to periodically evaluate the at least one database statistic, and to detect that the at least one database statistic exceeds a threshold value.
14. The distributed database system of claim 10 , wherein one or more processors are further configured to:
hash one or more column values in a row into a column value hash;
determine one hash range, out of a plurality of hash ranges, into which the column value hash falls; and
assign the row to one of the slices based on the determined hash range.
15. The distributed database system of claim 10 , wherein the at least one database statistic comprises, for each of the most common sets of one or more values for a set of one or more columns of the table, a total number of rows containing the set of one or more values for the set of one or more columns.
16. The distributed database system of claim 10 , wherein the at least one database statistic comprises a number of rows falling into each of a plurality of quantiles of potential values for a column.
17. The distributed database system of claim 10 , wherein the one or more processors are configured to split one of the slices into a plurality of new slices, and to detect the splitting of the slice into the plurality of new slices.
18. The distributed database system of claim 10 , wherein the database slices continue to be available for database transactions while the data in the first slice is redistributed to the new slices.
19. A computer-readable non-transitory storage medium comprising code capable of causing one or more processors to:
distribute a plurality rows of data unevenly across a plurality of slices of a table in a database according to a first hash function value of at least one column of each of the rows;
monitor at least one database statistic wherein the at least one database statistic comprises at least one of: a hash range occupancy ratio for each slice in said plurality of slices, a list of hot values for each column of each slice in said plurality of slices, a number of distinct values for each column of each slice in said plurality of slices, probabilistic data distribution of each column in each slice in said plurality of slices, or a quantile distribution of values for each column of each slice in said plurality of slices;
detect a redistribution condition based on the at least one monitored database statistic; and
respond to the detected redistribution condition by redistributing the rows of data across the slices according to a second hash function wherein the second hash function is based on a different number of column values of the table than the first hash function.
20. The medium of claim 19 , the medium further comprising code capable of causing one or more processors to:
hash one or more column values in a row into a column value hash;
determine one hash range, out of a plurality of hash ranges, into which the column value hash falls; and
assign the row to one of the slices based on the determined hash range.