Dividing a data partition based on a data storage coding scheme
A method for execution by at least one computing entity of a database system, the method includes obtaining a plurality of data partitions of a data set for storage in the database system, where the data set is organized in rows and columns, and the rows correspond to data records and the columns correspond to fields of the data records. The method further includes dividing a first partition of a plurality of data partitions to produce a first number of first raw data segments for storage in the database system, where the first number is based on a first data storage coding scheme. The method further includes dividing a second partition of the plurality of data partitions to produce a second number of second raw data segments for storage in the database system, where the second number is based on a second data storage coding scheme.
1 . A database system comprises:
a plurality of computing device clusters;
a parallelized data input sub-system including pluralities of loader nodes of pluralities of first computing devices of a first plurality of computing device clusters of the pluralities of computing device clusters, wherein a first computing device cluster of the first plurality of computing device clusters includes a first plurality of first computing devices of the pluralities of first computing devices, wherein a first computing device of the first plurality of first computing devices includes a first plurality of loader nodes of the pluralities of loader nodes, and wherein a set of loader nodes of the pluralities of loader nodes is operable to:
ingest a dataset, wherein the dataset includes rows of columnar data;
determine whether to partition the dataset based on size of the dataset and an analyzation determination; and
when it is determined to partition the dataset:
determine partitioning parameters for the dataset;
partition the dataset in accordance with the partitioning parameters to produce a plurality of data partitions;
divide the plurality of data partitions into sets of raw data segments in accordance with a plurality of segmenting factors, wherein a first data partition of the plurality of data partitions is divided into a first set of raw data segments of the sets of raw data segments;
organize the sets of raw data segments into pluralities of data slabs, wherein a first raw data segment of the first set of raw data segments is organized into a first plurality of data slabs of the pluralities of data slabs in accordance with a first key column to produce a first sorted data segment;
perform a redundancy function on the first sorted data segment to produce first parity data;
intersperse the first parity data with the first sorted data segment to produce a first sorted parity data segment; and
send the first sorted parity data segment to a first set of store and compute nodes of pluralities of store and compute nodes of pluralities of second computing devices of a second plurality of computing device clusters of the pluralities of computing device clusters for storage therein, wherein the second plurality of computing device clusters form a store and compute sub-system of the database system.
2 . The database system of claim 1 , wherein the set of loader nodes is further operable to:
when it is determined to not partition the dataset:
divide the dataset into a plurality of raw data segments in accordance with the plurality of segmenting factors;
organize the plurality of raw data segments into second pluralities of data slabs, wherein a second raw data segment of the plurality of raw data segments is organized into a second plurality of data slabs of the second pluralities of data slabs in accordance with a second key column to produce a second sorted data segment;
perform a second redundancy function on the second sorted data segment to produce second parity data;
intersperse the second parity data with the second sorted data segment to produce a second sorted parity data segment; and
send the second sorted parity data segment to a second set of store and compute nodes of the pluralities of store and compute nodes for storage therein.
3 . The database system of claim 1 , wherein the set of loader nodes is further operable to:
divide a second data partition of the plurality of data partitions into a second set of raw data segments of the sets of raw data segments, wherein the first set of raw data segments includes a first number of raw data segments, and wherein the second set of raw data segments includes a second number of raw data segments;
organize a second raw data segment of the second set of raw data segments into a second plurality of data slabs of the pluralities of data slabs in accordance with a second key column to produce a second sorted data segment;
perform the redundancy function on the second sorted data segment to produce second parity data;
intersperse the second parity data with the second sorted data segment to produce a second sorted parity data segment; and
send the second sorted parity data segment to the first set of store and compute nodes for storage therein.
4 . The database system of claim 3 , wherein the set of loader nodes is further operable to:
determine the first number based on first encoding parameters in accordance with the plurality of segmenting factors; and
determine the second number based on second encoding parameters in accordance with the plurality of segmenting factors.
5 . The database system of claim 3 , wherein the second number is equal to the first number.
6 . The database system of claim 3 , wherein the set of loader nodes is further operable to:
send the first sorted parity data segment for storage in a first one or more second computing devices of a first computing device cluster of the second plurality of computing device clusters; and
send the second parity sorted data segment for storage in a second one or more second computing devices of the first computing device cluster.
7 . The database system of claim 1 , wherein the plurality of segmenting factors include:
a data block size;
an available number of processing core resources of pluralities of processing core resources of the pluralities of store and compute nodes, wherein a store and compute node of the pluralities of store and compute nodes includes a plurality of processing core resources of the pluralities of processing core resources;
a number of the pluralities of store and compute nodes available;
a number of the pluralities of second computing devices available; and
a number of the second plurality of computing device clusters.
8 . The database system of claim 1 , wherein the set of loader nodes determines the analyzation determination based on one or more of:
a time of day; and
a predicted query for the data set.
9 . The database system of claim 1 , wherein the partitioning parameters comprise one or more of:
a data segment size;
a data block size;
memory formatting information;
network formatting information;
query processing information;
an available number of processing core resources of pluralities of processing core resources of the pluralities of store and compute nodes, wherein a store and compute node of the pluralities of store and compute nodes includes a plurality of processing core resources of the pluralities of processing core resources;
a number of the pluralities of store and compute nodes available;
a number of the pluralities of second computing devices available; and
a number of the second plurality of computing device clusters.
10 . A non-transitory computer readable storage medium comprises:
a first memory section that stores operational instructions that when executed by a set of loader nodes of pluralities of loader nodes of pluralities of first computing devices of a first plurality of computing device clusters of pluralities of computing device clusters of a database system, cause the set of loader nodes to:
ingest a dataset, wherein the dataset includes rows of columnar data, wherein the first plurality of computing device clusters form a parallelized data input sub-system of the database system;
determine whether to partition the dataset based on size of the dataset and an analyzation determination; and
when it is determined to partition the dataset:
determine partitioning parameters for the dataset;
partition the dataset in accordance with the partitioning parameters to produce a plurality of data partitions;
divide the plurality of data partitions into sets of raw data segments in accordance with a plurality of segmenting factors, wherein a first data partition of the plurality of data partitions is divided into a first set of raw data segments of the sets of raw data segments;
organize the sets of raw data segments into pluralities of data slabs, wherein a first raw data segment of the first set of raw data segments is organized into a first plurality of data slabs of the pluralities of data slabs in accordance with a first key column to produce a first sorted data segment;
perform a redundancy function on the first sorted data segment to produce first parity data;
intersperse the first parity data with the first sorted data segment to produce a first sorted parity data segment; and
send the first sorted parity data segment to a first set of store and compute nodes of pluralities of store and compute nodes of pluralities of second computing devices of a second plurality of computing device clusters of the pluralities of computing device clusters, wherein the second plurality of computing device clusters form a store and compute sub-system of the database system.
11 . The non-transitory computer readable storage medium of claim 10 , wherein the first memory section further stores operational instructions that when executed by the set of loader nodes, cause the set of loader nodes to:
when it is determined to not partition the dataset:
divide the dataset into a plurality of raw data segments in accordance with the plurality of segmenting factors;
organize the plurality of raw data segments into second pluralities of data slabs, wherein a second raw data segment of the plurality of raw data segments is organized into a second plurality of data slabs of the second pluralities of data slabs in accordance with a second key column to produce a second sorted data segment;
perform a second redundancy function on the second sorted data segment to produce second parity data;
intersperse the second parity data with the second sorted data segment to produce a second sorted parity data segment; and
send the second sorted parity data segment to a second set of store and compute nodes of the pluralities of store and compute nodes for storage therein.
12 . The non-transitory computer readable storage medium of claim 10 , wherein the first memory section further stores operational instructions that when executed by the set of loader nodes, cause the set of loader nodes to:
divide a second data partition of the plurality of data partitions into a second set of raw data segments of the sets of raw data segments, wherein the first set of raw data segments includes a first number of raw data segments, and wherein the second set of raw data segments includes a second number of raw data segments;
organize a second raw data segment of the second set of raw data segments into a second plurality of data slabs of the pluralities of data slabs in accordance with a second key column to produce a second sorted data segment;
perform the redundancy function on the second sorted data segment to produce second parity data;
intersperse the second parity data with the second sorted data segment to produce a second sorted parity data segment; and
send the second sorted parity data segment to the first set of store and compute nodes for storage therein.
13 . The non-transitory computer readable storage medium of claim 12 , wherein the first memory section further stores operational instructions that when executed by the set of loader nodes, cause the set of loader nodes to:
determine the first number based on first encoding parameters in accordance with the plurality of segmenting factors; and
determine the second number based on second encoding parameters in accordance with the plurality of segmenting factors.
14 . The non-transitory computer readable storage medium of claim 12 , wherein the second number is equal to the first number.
15 . The non-transitory computer readable storage medium of claim 12 , wherein the first memory section further stores operational instructions that when executed by the set of loader nodes, cause the set of loader nodes to:
send the first sorted parity data segment for storage in a first one or more second computing devices of a first computing device cluster of the second plurality of computing device clusters; and
send the second sorted parity data segment for storage in a second one or more second computing devices of the first computing device cluster.
16 . The non-transitory computer readable storage medium of claim 10 , wherein the plurality of segmenting factors include:
an available number of processing core resources of pluralities of processing core resources of the pluralities of store and compute nodes, wherein a store and compute node of the pluralities of store and compute nodes includes a plurality of processing core resources of the pluralities of processing core resources;
a number of the pluralities of store and compute nodes available;
a number of the pluralities of second computing devices available; and
a number of the second plurality of computing device clusters.
17 . The non-transitory computer readable storage medium of claim 10 , wherein the first memory section further stores operational instructions that when executed by the set of loader nodes, cause the set of loader nodes to determine the analyzation determination based on one or more of:
a time of day; and
a predicted query for the data set.
18 . The non-transitory computer readable storage medium of claim 10 , wherein the partitioning parameters comprise one or more of:
a data segment size;
a data block size;
memory formatting information;
network formatting information;
query processing information;
an available number of processing core resources of pluralities of processing core resources of the pluralities of store and compute nodes, wherein a store and compute node of the pluralities of store and compute nodes includes a plurality of processing core resources of the pluralities of processing core resources;
a number of the pluralities of store and compute nodes available;
a number of the pluralities of second computing devices available; and
a number of the second plurality of computing device clusters.