Methods and systems for increasing capacity and performing data rebalancing without downtime to a distributed shared-nothing database with serializable isolation

1. In a distributed shared-nothing database system maintaining data across a plurality of logical partitions each having a single thread of execution and being allocated to a position on a consistent hash ring, a computer-implemented method for transparently performing data rebalancing with serializable isolation when new logical partitions are added to the database system, the method comprising the steps of:

(a) calculating a desired consistent hash ring including an added new partition to the database system in one or more positions in the consistent hash ring previously allocated to one or more pre-existing partitions;

(b) generating an index identifying data to be moved from said one or more pre-existing partitions to the new partition;

(c) executing a series of rebalancing transactions in serialized transaction queues on said one or more pre-existing partitions and the new partition to transfer different portions of the data to be moved from said one or more pre-existing partitions to the new partition, while maintaining serializable isolation, and updating the consistent hash ring after each rebalancing transaction has been executed to reflect transfer of the data from said one or more pre-existing partitions to the new partition, wherein serializable isolation is maintained by pausing execution of pending transactions for at least one of the one or more pre-existing partitions and the new partition for a specified amount of time, the specified amount of time representing an amount of time expected to transfer data from the one or more pre-existing partitions to the new partition; and

(d) rerouting in-flight transactions to the new partition for data moved from the one or more pre-existing partitions to the new partition.

2. The method of claim 1 , wherein the database system comprises one or more database nodes, each including one or more partitions, and a rebalance coordinator node connected in a computer network for performing rebalancing among the partitions.

3. The method of claim 1 , wherein step (b) comprises initiating a background scan to materialize the index on each table in the database containing rows to be moved across ranges of data.

4. The method of claim 3 , further comprising selecting one of the ranges to be moved, and selecting a portion of the range based on an expected amount of time to transfer data in the portion of the range.

5. The method of claim 4 , wherein the amount of time is specified by a user.

6. The method of claim 1 , further comprising updating the index as part of regular transaction execution.

7. The method of claim 1 , wherein step (d) comprises sending the in-flight transactions to a routing mesh to be rerouted in accordance with the updated consistent hash ring.

8. The method of claim 1 , further comprising lazily propagating updates to the consistent hash ring after step (d).

9. A distributed shared-nothing database system maintaining data across a plurality of logical partitions each having a single thread of execution and being allocated to a position on a consistent hash ring, the system comprising:

at least one processor;

memory associated with the at least one processor; and

a program supported in the memory for transparently performing data rebalancing with serializable isolation when new logical partitions are added to the database system, the program containing a plurality of instructions which, when executed by the at least one processor, cause the at least one processor to:

(a) calculate a desired consistent hash ring including an added new partition to the database system in one or more positions in the consistent hash ring previously allocated to one or more pre-existing partitions;

(b) generate an index identifying data to be moved from said one or more pre-existing partitions to the new partition;

(c) execute a series of rebalancing transactions in serialized transaction queues on said one or more pre-existing partitions and the new partition to transfer different portions of the data to be moved from said one or more pre-existing partitions to the new partition, while maintaining serializable isolation, and update the consistent hash ring after each rebalancing transaction has been executed to reflect transfer of the data from said one or more pre-existing partitions to the new partition, wherein serializable isolation is maintained by pausing execution of pending transactions for at least one of the one or more pre-existing partitions and the new partition for a specified amount of time, the specified amount of time representing an amount of time expected to transfer data from the one or more pre-existing partitions to the new partition; and

(d) reroute in-flight transactions to the new partition for data moved from the one or more pre-existing partitions to the new partition.

10. The database system of claim 9 , wherein the database system comprises one or more database nodes, each including one or more partitions, and a rebalance coordinator node connected in a computer network for performing rebalancing among the partitions.

11. The database system of claim 9 , wherein (b) comprises initiating a background scan to materialize the index on each table in the database containing rows to be moved across ranges of data.

12. The database system of claim 11 , further comprising instructions for selecting one of the ranges to be moved, and selecting a portion of the range based on an expected amount of time to transfer data in the portion of the range.

13. The database system of claim 12 , wherein the amount of time is specified by a user.

14. The database system of claim 9 , further comprising instructions for updating the index as part of regular transaction execution.

15. The database system of claim 9 , wherein (d) comprises sending the in-flight transactions to a routing mesh to be rerouted in accordance with the updated consistent hash ring.

16. The database system of claim 9 , further comprising instructions for lazily propagating updates to the consistent hash ring after step (d).