Splitting and moving ranges in a distributed system

1. A computer-implemented method for a distributed transaction in a distributed database system executed by one or more processors, the method comprising:

identifying a request to insert a split point in a source group, the source group comprising one or more tablet replicas, each tablet comprises a list of immutable files or layers including at least a portion of data from a table in the distributed database system, and the split point splitting the data in the source group into a first range and a second range different than the first range;

in response to the request to insert the split point in the source group:

sending a list of filenames in the first range of the source group to a first target group of a plurality of target groups, the first target group comprising one or more tablet replicas;

creating, at the first target group, a virtual copy of files represented by the list of filenames in the first range, the virtual copy making data of the files available, each using a new name, without duplicating the data of the files;

applying a distributed transaction including the source group and the target groups, wherein the transaction updates metadata tables to reflect new data locations, split points and metadata split points in the source group and target groups to reflect the move, and wherein the transaction further comprises range data for the source group and target group, wherein the range data specifies ranges of user data to be moved;

applying the range data by a source tablet of the source group, preserving a current layer stack and preparing the current layer stack for being served to a target tablet of the first target group;

retrieving the current layer stack of the source tablet and inserting one or more proxy layers in an apply loop which specify the ranges to be moved; and

writing contents of all ranges that have been moved into a replicated table in the first target group before deleting the source group.

2. The method of claim 1 , comprising:

adding the files in the first range to the first target group;

instructing the source group that the first target group no longer depends on the files in the first range at the source group; and

deleting the files in the first range from the source group.

3. The method of claim 1 , comprising:

in response to the request to insert the split point in the source group:

sending a list of filenames in the second range of the source group to a second target group, the second target group comprising one or more tablet replicas; and

creating, at the second target group, a virtual copy of files represented by the list of filenames in the second range, the virtual copy making the contents of the files available each under a new name without duplicating data of the files.

4. The method of claim 1 , comprising:

sending, from the source group, contents of in-memory buffered updates in the first range; and

adding, at the first target group, the contents of in-memory buffered updates in the first range from the source group to the in-memory buffered updates of the first target group.

5. The method of claim 1 , wherein the distributed transaction is executed according to a two-phase commit protocol comprising a voting phase and a commit phase, wherein applying the distributed transaction occurs after the voting phase and during the commit phase.

6. The method of claim 5 , wherein a majority of tablets in each group must commit in the voting phase for the distributed transaction to complete.

7. The method of claim 5 , wherein, in response to a vote to abort in the voting phase, each group undoes the transaction.

8. A system for executing a distributed transaction in a distributed database, the system comprising:

memory for storing data; and

one or more processors operable to access the memory and perform operations comprising:

identifying a request to insert a split point in a source group, the source group comprising one or more tablet replicas, each tablet comprises a list of immutable files or layers including at least a portion of data from a table in the distributed database system, and the split point splitting the data in the source group into a first range and a second range different than the first range;

in response to the request to insert the split point in the source group:

sending a list of filenames in the first range of the source group to a first target group of a plurality of target groups, the first target group comprising one or more tablet replicas; and

creating, at the first target group, a virtual copy of files represented by the list of filenames in the first range, the virtual copy making data of the files available, each using a new name, without duplicating the data of the files;

applying a distributed transaction including the source group and the target groups, wherein the transaction updates metadata tables to reflect new data locations, split points and metadata split points in the source group and target groups to reflect the move, and wherein the transaction further comprises range data for the source group and target groups, wherein the range data specifies the ranges of user data to be moved;

applying the range data by a source tablet of the source group, preserving a current layer stack and preparing the current layer stack for being served to a target tablet of the first target group;

retrieving the current layer stack of the source tablet and inserting one or more proxy layers in an apply loop which specify the ranges to be moved; and

writing contents of all ranges that have been moved into a replicated table in the first target group before deleting the source group.

9. The system of claim 8 , wherein the operations comprise:

adding the files in the first range to the first target group;

instructing the source group that the first target group no longer depends on the files in the first range at the source group; and

deleting the files in the first range from the source group.

10. The system of claim 8 , wherein the operations comprise:

in response to the request to insert the split point in the source group:

sending a list of filenames in the second range of the source group to a second target group, the second target group comprising one or more tablet replicas; and

creating, at the second target group, a virtual copy of files represented by the list of filenames in the second range, the virtual copy making the contents of the files available each under a new name without duplicating data of the files.

11. The system of claim 8 , wherein the operations comprise:

sending, from the source group, contents of in-memory buffered updates in the first range; and

adding, at the first target group, the contents of in-memory buffered updates in the first range from the source group to the in-memory buffered updates of the first target group.

12. The system of claim 8 , wherein the distributed transaction is executed according to a two-phase commit protocol comprising a voting phase and a commit phase.

13. The system of claim 12 , wherein a majority of tablets in each group must commit for the distributed transaction to complete.

14. The system of claim 12 , wherein, in response to a vote to abort in the voting phase, each group undoes the transaction.

15. A non-transitory, computer-readable medium storing instructions operable when executed to cause at least one processor to perform operations for executing a distributed transaction in a distributed database, the operations comprising:

identifying a request to insert a split point in a source group, the source group comprising one or more tablet replicas, each tablet comprises a list of immutable files or layers including at least a portion of data from a table in the distributed database system, and the split point splitting the data in the source group into a first range and a second range different than the first range;

in response to the request to insert the split point in the source group:

sending a list of filenames in the first range of the source group to a first target group of a plurality of target groups, the first target group comprising one or more tablet replicas; and

creating, at the first target group, a virtual copy of files represented by the list of filenames in the first range, the virtual copy making data of the files available, each using a new name, without duplicating the data of the files;

applying a distributed transaction including the source group and the target groups, wherein the transaction updates metadata tables to reflect new data locations, split points and metadata split points in the source group and target groups to reflect the move, and wherein the transaction further comprises range data for the source group and target groups, wherein the range data specifies the ranges of user data to be moved;

applying the range data by a source tablet of the source group, preserving a current layer stack and preparing the current layer stack for being served to a target tablet of the first target group;

retrieving the current layer stack of the source tablet and inserting one or more proxy layers in an apply loop which specify the ranges to be moved; and

writing contents of all ranges that have been moved into a replicated table in the first target group before deleting the source group.

16. The computer-readable medium of claim 15 , wherein the operations comprise:

adding the files in the first range to the first target group;

instructing the source group that the first target group no longer depends on the files in the first range at the source group; and

deleting the files in the first range from the source group.

17. The computer-readable medium of claim 15 , wherein the operations comprise:

in response to the request to insert the split point in the source group:

sending a list of filenames in the second range of the source group to a second target group, the second target group comprising one or more tablet replicas; and

creating, at the second target group, a virtual copy of files represented by the list of filenames in the second range, the virtual copy making the contents of the files available each under a new name without duplicating data of the files.

18. The computer-readable medium of claim 15 , wherein the operations comprise:

sending, from the source group, contents of in-memory buffered updates in the first range; and

adding, at the first target group, the contents of in-memory buffered updates in the first range from the source group to the in-memory buffered updates of the first target group.

19. The computer-readable medium of claim 15 , wherein the distributed transaction is executed according to a two-phase commit protocol comprising a voting phase and a commit phase.

20. The computer-readable medium of claim 19 , wherein a majority of tablets in each group must commit for the distributed transaction to complete, and

if the voting phase includes a vote to abort, each group undoes the transaction.