Unbalanced partitioning of database for application data
Provided is a database system and method in which storage is partitioned in an unbalanced format for faster access. In one example, the method may include one or more of receiving a request to store a data record, identifying a partition from among a plurality of partitions of a database based on a shard identifier in the request, automatically determining a unique range of data identifiers designated to the partition from the plurality of partitions, respectively, based on an unbalanced partitioning, determining whether the data identifier is available within the unique range of data identifiers of the identified partition, and storing the data record at the identified partition in response to determining the data identifier is available. The unbalanced partitioning according to various embodiments reduces the partitions that need to be checked during a data insert/access operation of the database.
1. A computing system comprising:
a data store; and
a processor configured to
map a plurality of shard identifiers corresponding to a plurality of partitions of a database to a plurality of columns within a two-dimensional data structure stored in the data store and map a plurality of non-overlapping ranges of data identifiers to a plurality of rows in the two-dimensional data structure in the data store, where each shard identifier among the plurality of shard identifiers is designated a different non-overlapping range of data identifiers within the two-dimensional arrangement; and
receive a request to store a data record comprising a shard identifier and a data identifier assigned by the database to the data record,
determine that a partition exists for the shard identifier from table metadata of the plurality of partitions,
in response to determining that the partition exists for the shard identifier, identify a column within the two-dimensional data structure which is mapped to the shard identifier based on the mapping of the plurality of shard identifiers to the plurality of columns within the two-dimensional data structure and identify a row with a range of data identifiers designated to the partition based on the mapping of the plurality of different non-overlapping ranges of data identifiers to the plurality of rows within the two-dimensional data structure, and
execute a unicity check for the data identifier in the request on only the non-overlapping range of data identifiers designated to the partition within the identified row of the two-dimensional data structure to determine whether the data identifier is available within the partition,
wherein the processor is further configured to store the data record at the partition in response to determining the data identifier is available.
2. The computing system of claim 1 , wherein the processor is further configured to perform the unicity check on only one database node corresponding to the shard identifier from among a plurality of database nodes corresponding to the plurality of shard identifiers to determine if the data identifier is already assigned to another data record stored in the partition.
3. The computing system of claim 1 , wherein the two-dimensional data structure comprises a two-dimensional array of cells that correspond to the plurality of rows of non-overlapping ranges of data identifiers designated to the plurality of columns corresponding to the plurality of shard identifiers.
4. The computing system of claim 3 , wherein the processor is configured to assign the plurality of non-overlapping ranges of data identifiers to a plurality of different shard identifiers to create a diagonal pattern of cells with mapped non-overlapping ranges of data identifiers in the two-dimensional data structure.
5. The computing system of claim 3 , wherein ranges of data identifiers that are designated to shard identifiers of other partitions among the plurality of partitions are not created for the shard identifier within the two-dimensional arrangement.
6. The computing system of claim 1 , wherein the plurality of partitions are stored on a plurality of database nodes respectively, with each database node corresponding to a unique shard identifier and storing only one partition.
7. The computing system of claim 6 , wherein a first database node from among the plurality of database nodes comprises a different type of storage structure than a second database node from among the plurality of database nodes.
8. The computing system of claim 1 , wherein the data identifier comprises a next available data identifier within the non-overlapping range of data identifiers designated to the partition.
9. A method comprising:
mapping a plurality of shard identifiers corresponding to a plurality of partitions of a database to a plurality of columns within a two-dimensional data structure stored in a data store and mapping a plurality of non-overlapping ranges of data identifiers to a plurality of rows in the two-dimensional data structure in the data store, where each shard identifier among the plurality of shard identifiers is designated a different non-overlapping range of data identifiers within the two-dimensional arrangement;
receiving a request to store a data record, where the request comprises a shard identifier and a data identifier assigned by the database to the data record;
determining that a partition exists for the shard identifier from table metadata of the plurality of partitions;
in response to determining that the partition exists for the shard identifier, identifying a column within the two-dimensional data structure which is mapped to the shard identifier based on the mapping of the plurality of shard identifiers to the plurality of columns within the two-dimensional data structure and identifying a row of a range of data identifiers designated to the partition based on the mapping of the plurality of different non-overlapping ranges of data identifiers to the plurality of rows within the two-dimensional data structure;
executing a unicity check for the data identifier in the request on only the non-overlapping range of data identifiers designated to the partition within the identified row of the two-dimensional data structure to determine that the data identifier is available within the partition; and
storing the data record at the partition in response to determining the data identifier is available.
10. The method of claim 9 , wherein the performing comprises performing the unicity check on only one database node corresponding to the shard identifier from among a plurality of database nodes corresponding to the plurality of shard identifiers to determine if the data identifier is already assigned to another data record stored in the partition.
11. The method of claim 9 , wherein the two-dimensional data structure comprises a two-dimensional array of cells that correspond to the plurality of rows of non-overlapping ranges of data identifiers designated to the plurality of columns corresponding to the plurality of shard identifiers.
12. The method of claim 11 , wherein the mapping comprises assigning the plurality of non-overlapping ranges of data identifiers to a plurality of different shard identifiers to create a diagonal pattern of cells with mapped non-overlapping ranges of data identifiers in the two-dimensional data structure.
13. The method of claim 11 , wherein ranges of data identifiers that are designated to shard identifiers of other partitions among the plurality of partitions are not created for the shard identifier within the two-dimensional arrangement.
14. The method of claim 9 , wherein the plurality of partitions are stored on a plurality of database nodes respectively, with each database node corresponding to a unique shard identifier and storing only one partition.
15. The method of claim 14 , wherein a first database node from among the plurality of database nodes comprises a different type of storage structure than a second database node from among the plurality of database nodes.
16. The method of claim 9 , wherein the data identifier comprises a next available data identifier within the non-overlapping range of data identifiers designated to the partition.
17. A non-transitory computer-readable storage medium storing program instructions that when executed cause a processor to perform a method comprising:
mapping a plurality of shard identifiers corresponding to a plurality of partitions of a database to a plurality of columns within a two-dimensional data structure stored in a data store and mapping a plurality of non-overlapping ranges of data identifiers to a plurality of rows in the two-dimensional data structure in the data store, where each shard identifier among the plurality of shard identifiers is designated a different non-overlapping range of data identifiers within the two-dimensional arrangement;
receiving a request to store a data record, where the request comprises a shard identifier and a data identifier assigned by the database to the data record;
determining that a partition exists for the shard identifier from table metadata of the plurality of partitions;
in response to determining that the partition exists for the shard identifier, identifying a column within the two-dimensional data structure which is mapped to the shard identifier based on the mapping of the plurality of shard identifiers to the plurality of columns within the two-dimensional data structure and identifying a row of a range of data identifiers designated to the partition based on the mapping of the plurality of different non-overlapping ranges of data identifiers to the plurality of rows within the two-dimensional data structure;
executing a unicity check for the data identifier in the request on only the non-overlapping range of data identifiers designated to the partition within the identified row of the two-dimensional data structure to determine that the data identifier is available within the partition; and
storing the data record at the partition in response to determining the data identifier is available.
18. The non-transitory computer readable medium of claim 17 , wherein the performing comprises performing the unicity check on only one database node corresponding to the shard identifier from among a plurality of database nodes corresponding to the plurality of partitions to determine if the data identifier is already assigned to another data record stored in the partition.
19. The non-transitory computer readable medium of claim 17 , wherein the two-dimensional data structure comprises a two-dimensional array of cells that correspond to the plurality of rows of non-overlapping ranges of data identifiers designated to the plurality of columns corresponding to the plurality of shard identifiers.
20. The non-transitory computer readable medium of claim 19 , wherein the mapping comprises assigning the plurality of non-overlapping ranges of data identifiers to a plurality of different shard identifiers to create a diagonal pattern of cells with mapped non-overlapping ranges of data identifiers in the two-dimensional data structure.