Cosharding and randomized cosharding
The technology relates to cosharding tables within a distributed storage system. A data table including one or more rows may be received. Each row in the data table may include an identifier key and pieces of data. Each piece of data in the data table may be indexed into individual rows of an index table, wherein each row in the index table includes data associated with the identifier key of the data table from which the piece of data in the respective row was indexed. The index table may be sharded into splits, wherein the sharding includes assigning each row of the index table into one of the splits based on the identifier key of the data table from which the piece of data in the respective row was indexed. The splits may be more stored into two or more or more portions of the distributed storage system.
1 . A method for cosharding tables within a distributed storage system, the method comprising:
receiving, by one or more processors, a data table including one or more rows, wherein each row includes:
an identifier key having a value, and
one or more pieces of data;
for each row of the one or more rows of the data table, indexing, by the one or more processors, each piece of the one or more pieces of data into an individual row of one of a plurality of index tables stored on separate portions of the distributed storage system, such that each row in the plurality of index tables includes a piece of the one or more pieces of data and the identifier key of the row of the data table from which the piece of data was indexed; and
sharding, by the one or more processors, each of the plurality of index tables into splits, wherein the sharding includes assigning each row of the plurality of index tables into a respective one of the splits based on the value of the identifier key of the respective row.
2 . The method of claim 1 , further comprising:
sharding each row of the one or more rows of the data table, such that each row is together with the individual row of the index table to which the piece of data of the respective row of the data table was indexed.
3 . The method of claim 1 , wherein each row includes two or more identifier keys.
4 . The method of claim 3 , wherein and each of the two or more identifier keys has a respective value.
5 . The method of claim 4 , wherein the sharding of each of the plurality of index tables into splits further comprises assigning each row of the plurality of index tables into a respective one of the splits based on the value of each of the two or more identifier keys of the data table from which the piece of data in the respective row was indexed.
6 . The method of claim 4 , wherein the sharding of each of the plurality of index tables into splits further comprises assigning each row of the plurality of index tables into a respective one of the splits based on the value of one of the two or more identifier keys of the data table from which the piece of data in the respective row was indexed.
7 . The method of claim 1 , wherein the value of the identifier key in each row of the one or more rows is a timestamp.
8 . A system for cosharding a table, the system comprising:
a distributed storage system; and
one or more processors, wherein the one or more processors are configured to:
receive a data table including one or more rows, wherein each row includes:
an identifier key having a value, and
one or more pieces of data;
for each row of the one or more rows of the data table, index each piece of the one or more pieces of data into an individual row of one of a plurality of index tables stored on separate portions of the distributed storage system, such that each row in the plurality of index tables includes a piece of the one or more pieces of data and the identifier key of the row of the data table from which the piece of data was indexed; and
shard each of the plurality of index tables into splits, wherein the sharding includes assigning each row of the plurality of index tables into a respective one of the splits based on the value of the identifier key of the respective row.
9 . The system of claim 8 , wherein the one or more processors are further configured to:
shard each row of the one or more rows of the data table, such that each row is together with the individual row of the index table to which the piece of data of the respective row of the data table was indexed.
10 . The system of claim 8 , wherein each row includes two or more identifier keys.
11 . The system of claim 8 , wherein and each of the two or more identifier keys has a respective value.
12 . The system of claim 11 , wherein the sharding of each of the plurality of index tables into splits further comprises assigning each row of the plurality of index tables into a respective one of the splits based on the value of each of the two or more identifier keys of the data table from which the piece of data in the respective row was indexed.
13 . The system of claim 11 , wherein the sharding of each of the plurality of index tables into splits further comprises assigning each row of the plurality of index tables into a respective one of the splits based on the value of one of the two or more identifier keys of the data table from which the piece of data in the respective row was indexed.
14 . The system of claim 8 , wherein the value of the identifier key in each row of the one or more rows is a timestamp.
15 . A non-transitory computer-readable medium storing instructions, which when executed by one or more processors, cause the one or more processors to:
receive a data table including one or more rows, wherein each row includes:
an identifier key having a value, and
one or more pieces of data;
for each row of the one or more rows of the data table, index each piece of the one or more pieces of data in the data table into an individual row of one of a plurality of index tables stored on separate portions of the distributed storage system, such that each row in the plurality of index tables includes a piece of the one or more pieces of data and the identifier key of the row of the data table from which the piece of data was indexed; and
shard each of the plurality of index tables into splits, wherein the sharding includes assigning each row of the plurality of index tables into a respective one of the splits based on the value of the identifier key of the respective row.
16 . The non-transitory computer-readable medium of claim 15 , wherein the instructions, further cause the one or more processors to:
shard each row of the one or more rows of the data table, such that each row is together with the individual row of the index table to which the piece of data of the respective row of the data table was indexed.
17 . The non-transitory computer-readable medium of claim 15 , wherein each row includes two or more identifier keys.
18 . The non-transitory computer-readable medium of claim 15 , wherein and each of the two or more identifier keys has a respective value.
19 . The non-transitory computer-readable medium of claim 18 , wherein the sharding of each of the plurality of index tables into splits further comprises assigning each row of the plurality of index tables into a respective one of the splits based on the value of each of the two or more identifier keys of the data table from which the piece of data in the respective row was indexed.
20 . The non-transitory computer-readable medium of claim 18 , wherein the sharding of each of the plurality of index tables into splits further comprises assigning each row of the plurality of index tables into a respective one of the splits based on the value of one of the two or more identifier keys of the data table from which the piece of data in the respective row was indexed.