Cosharding and randomized cosharding
The technology relates to cosharding tables within a distributed storage system. A data table including one or more rows may be received. Each row in the data table may include an identifier key and pieces of data. Each piece of data in the data table may be indexed into individual rows of an index table, wherein each row in the index table includes data associated with the identifier key of the data table from which the piece of data in the respective row was indexed. The index table may be sharded into splits, wherein the sharding includes assigning each row of the index table into one of the splits based on the identifier key of the data table from which the piece of data in the respective row was indexed. The splits may be stored into two or more portions of the distributed storage system.
1. A method for cosharding tables within a distributed storage system, the method comprising:
receiving, by one or more processors, a data table including one or more rows, wherein each row includes one or more pieces of data and an identifier key assigned to each piece of data, wherein each identifier key is a randomly generated value within a specified range of values;
indexing, by the one or more processors, each piece of the one or more pieces of data in the data table into an individual row of one of a plurality of index tables stored on separate portions of the distributed storage system, wherein each row in the plurality of index tables includes a piece of the one or more pieces of data and the identifier key of the row of the data table from which the piece of data was indexed;
sharding, by the one or more processors, each of the plurality of index tables into splits, wherein the sharding includes assigning each row of the plurality of index tables into a respective one of the splits based on the identifier key of the data table from which the piece of data in the respective row was indexed; and
sharding each row of the one or more rows of the data table, such that each row is together with the individual row of the index table to which the piece of data of the respective row of the data table was indexed.
2. The method of claim 1 , wherein each of the separate portions of the distributed storage system is assigned a range subset within the specified range of values.
3. The method of claim 1 , wherein indexing each piece of the one or more pieces of data includes, for each piece of the one or more pieces of data in the data table:
comparing the identifier assigned to the piece of data to the range subsets to determine a portion of the portions of the distributed storage system assigned a range subset covering the identifier assigned to the piece of data; and
indexing the piece of data to an index table stored on the portion of the distributed storage system assigned the range subset covering the identifier assigned to the piece of data.
4. The method of claim 1 , wherein the sharded one or more rows from the data table are stored in the same split as the individual row of the index table to which the pieces of data of the respective row of the data table were indexed.
5. The method of claim 1 , wherein the data in the data table is in one or more columns of the data table.
6. The method of claim 1 , wherein the identifier key further includes a timestamp, and the splits are sorted by the timestamp.
7. A system for cosharding a table, the system comprising:
a distributed storage system; and
one or more processors, wherein the one or more processors are configured to:
receive a data table including one or more rows, wherein each row includes one or more pieces of data and an identifier key assigned to each piece of data, wherein each identifier key is a randomly generated value within a specified range of values;
index each piece of the one or more pieces of data in the data table into an individual row of one of a plurality of index tables stored on separate portions of the distributed storage system, wherein each row in the plurality of index tables includes a piece of the one or more pieces of data and the identifier key of the row of the data table from which the piece of data was indexed;
shard each of the plurality of index tables into splits, wherein the sharding includes assigning each row of the plurality of index tables into a respective one of the splits based on the identifier key of the data table from which the piece of data in the respective row was indexed; and
shard each row of the one or more rows of the data table, such that each row is together with the individual row of the index table to which the piece of data of the respective row of the data table was indexed.
8. The system of claim 7 , wherein each of the separate portions of the distributed storage system is assigned a range subset within the specified range of values.
9. The system of claim 7 , wherein indexing each piece of the one or more pieces of data includes, for each piece of the one or more pieces of data in the data table:
comparing the identifier assigned to the piece of data to the range subsets to determine a portion of the portions of the distributed storage system assigned a range subset covering the identifier assigned to the piece of data; and
indexing the piece of data to an index table stored on the portion of the distributed storage system assigned the range subset covering the identifier assigned to the piece of data.
10. The system of claim 7 , wherein the sharded one or more rows from the data table are stored in the same split as the individual row of the index table to which the pieces of data of the respective row of the data table were indexed.
11. The system of claim 7 , wherein the data in the data table is in one or more columns of the data table.
12. The system of claim 7 , wherein the identifier key further includes a timestamp, and the splits are sorted by the timestamp.
13. A non-transitory computer-readable medium storing instructions, which when executed by one or more processors, cause the one or more processors to:
receive a data table including one or more rows, wherein each row includes one or more pieces of data and an identifier key assigned to each piece of data, wherein each identifier key is a randomly generated value within a specified range of values;
index each piece of the one or more pieces of data in the data table into an individual row of one of a plurality of index tables stored on separate portions of a distributed storage system, wherein each row in the plurality of index tables includes a piece of the one or more pieces of data and the identifier key of the row of the data table from which the piece of data was indexed;
shard each of the plurality of index tables into splits, wherein the sharding includes assigning each row of the plurality of index tables into a respective one of the splits based on the identifier key of the data table from which the piece of data in the respective row was indexed; and
shard each row of the one or more rows of the data table, such that each row is together with the individual row of the index table to which the piece of data of the respective row of the data table was indexed.
14. The non-transitory computer-readable medium of claim 13 , wherein each of the separate portions of the distributed storage system is assigned a range subset within the specified range of values.
15. The non-transitory computer-readable medium of claim 13 , wherein indexing each piece of the one or more pieces of data includes, for each piece of the one or more pieces of data in the data table:
comparing the identifier assigned to the piece of data to the range subsets to determine a portion of the portions of the distributed storage system assigned a range subset covering the identifier assigned to the piece of data; and
indexing the piece of data to an index table stored on the portion of the distributed storage system assigned the range subset covering the identifier assigned to the piece of data.
16. The non-transitory computer-readable medium of claim 13 , wherein the sharded one or more rows from the data table are stored in the same split as the individual row of the index table to which the pieces of data of the respective row of the data table were indexed.
17. The non-transitory computer-readable medium of claim 13 , wherein the data in the data table is in one or more columns of the data table.