Processing row data via a plurality of processing core resources
A database system operates to: receive, via a communication network, a record stream from a data source that includes a plurality of row data, wherein each of the plurality of row data includes a set of records for storage as relational database rows of at least one database table and further indicates a batch number corresponding to the set of records; add each of the plurality of row data to a pending row data pool; and perform parallelized generation of a plurality of pages from the plurality of row data to each include multiple records of the plurality of row data via a plurality of processing core resources based on each processing core resource in the plurality of processing core resources including power, processing a corresponding subset of the plurality of row data, independently from and in parallel with processing of other subsets of the plurality of row data via other ones of the plurality of processing core resources.
1 . A database system comprising:
a data input sub-system including a plurality of computing device clusters, wherein a computing device cluster of the plurality of computing device clusters includes a plurality of computing devices, wherein a computing device of the plurality of computing devices includes a plurality of computing nodes, wherein a computing node of the plurality of computing nodes includes:
main memory that is operably coupled to:
receive a plurality of row data of a sub-segment of a segment of a segment group of a partition of a dataset, wherein the dataset includes a plurality of rows of columnar data, where columnar data includes a plurality of columns of data, wherein the sub-segment includes a sub-segment set of rows of columnar data of the plurality of rows of the columnar data, and wherein a row data of the plurality of row data includes a plurality of portions of the sub-segment set of rows of columnar data, wherein the dataset includes at least ten-thousand rows of data; and
store the plurality of row data in a database memory section of the main memory; and
a plurality of processing core resources, wherein a first processing core resource of the plurality of core resources is operably coupled to:
retrieve first row data of the plurality of row data from the database memory section when the first row of data is available;
process the first row data for inclusion in a first division of a plurality of divisions of the sub-segment;
when the first processing core resource has completed processing the first row data, determine whether another row data of the plurality of row data is available; and
when the other data row is available, processing the other row data for inclusion in the first division of the sub-segment, wherein, after the plurality of row data has been stored in the main memory, the plurality of processing core resources has processed the plurality of row data to produce the plurality of divisions of the sub-segment.
2 . The database system of claim 1 , wherein the dataset further comprises:
a plurality of partitions, including the partition, wherein the partition includes:
a partition set of rows of columnar data of the plurality of rows of the columnar data of the dataset;
one or more segment groups, including the segment group, wherein the segment group includes:
a segment group set of rows of columnar data of the partition set of rows of columnar data; and
a group of segments, including the segment, wherein the segment includes:
a segment set of rows of columnar data of the segment group set of rows of columnar data;
the segment set of rows of columnar data includes a group of sub-segments of rows of columnar data, wherein the group of sub-segments includes the sub-segment, which includes:
a sub-segment set of rows of columnar data of the segment set of rows of columnar data; and
the sub-segment set of rows of columnar data includes a group of divisions of sub-segment sets, wherein the group of divisions of sub-segment sets includes the division of the sub-segment.
3 . The database system of claim 1 , wherein the first processing core resource comprises:
a memory device for storing the first division of the sub-segment.
4 . The database system of claim 1 , wherein each processing core resource of the plurality of processing core resources is further operable to:
retrieve and process row data of the plurality of row data independently of the other processing core resources; and
store a respective division of the plurality of divisions of the sub-segment.
5 . The database system of claim 1 , wherein the database memory section comprises:
a logical queue that preserves row ordering of a sub-segment set of rows of columnar data of the plurality of rows of columnar data of the sub-segment to support deduplication during or after page generation.
6 . The database system of claim 1 , wherein a row data of the plurality of row data comprises: source identification information and is queued in a source-specific order.
7 . The database system of claim 1 , wherein processing of the row data comprises one or more of:
generating a complete page;
generating a portion of a page;
generating page metadata;
generating index data; and
compressing data values.
8 . A computer readable memory device comprises:
a first memory that stores operational instructions that, when executed by a computing node of a plurality of computing nodes of a computing device of a plurality of computing devices of a computing device cluster of a plurality of computing device clusters of a data input sub-system of a database system, cause the computing node to:
receive a plurality of row data of a sub-segment of a segment of a segment group of a partition of a dataset, wherein the dataset includes a plurality of rows of columnar data, where columnar data includes a plurality of columns of data, wherein the sub-segment includes a sub-segment set of rows of columnar data of the plurality of rows of the columnar data, and wherein a row data of the plurality of row data includes a plurality of portions of the sub-segment set of rows of columnar data, wherein the dataset includes at least ten-thousand rows of data; and
store the plurality of row data of a sub-segment in a database memory section of main memory of the computing node;
a second memory that stores operational instructions that, when executed by a first processing core resource of a plurality of processing core resource of the computing node causes the first processing core resource to:
retrieve first row data of the plurality of row data from the database memory section when the first row data is available;
process the first row data inclusion in a first division of a plurality of divisions of the sub-segment;
when the first processing core resource has completed processing the first row data, determine whether another row data of the plurality of row data is available; and
when the other data row is available, processing the other row data for inclusion in the first division of the sub-segment, wherein, after the plurality of row data has been stored in the main memory, the plurality of processing core resources have processed the plurality of row data to produce the plurality of divisions of the sub-segment.
9 . The computer readable memory of claim 8 , wherein the dataset further comprises:
a plurality of partitions, including the partition, wherein the partition includes:
a partition set of rows of columnar data of the plurality of rows of the columnar data of the dataset;
one or more segment groups, including the segment group, wherein the segment group includes:
a segment group set of rows of columnar data of the partition set of rows of columnar data; and
a group of segments, including the segment, wherein the segment includes:
a segment set of rows of columnar data of the segment group set of rows of columnar data;
the segment set of rows of columnar data includes a group of sub-segments of rows of columnar data, wherein the group of sub-segments includes the sub-segment, which includes:
a sub-segment set of rows of columnar data of the segment set of rows of columnar data; and
the sub-segment set of rows of columnar data includes a group of divisions of sub-segment sets, wherein the group of divisions of sub-segment sets includes the division of the sub-segment.
10 . The computer readable memory of claim 8 , wherein the second memory further stores operational instruction that, when executed by the first processing core resource, causes the first processing core resource to:
retrieve and process row data of the plurality of row data independently of the other processing core resources; and
store a respective division of the plurality of divisions of the sub-segment.
11 . The computer readable memory of claim 8 , wherein the first memory section further stores operational instructions that, when executed by a computing node, cause the computing node to:
establish a logical queue within the database memory section, wherein the logical queue preserves row ordering of a sub-segment set of rows of columnar data of the plurality of rows of columnar data of the sub-segment to support deduplication during or after page generation.
12 . The computer readable memory of claim 8 , wherein a row data of the plurality of row data comprises: source identification information and is queued in a source-specific order.
13 . The computer readable memory of claim 8 , wherein the second memory further stores operational instructions that, when executed by the first processing core resource, causes the first processing core resource to process the first row data by one or more of:
generating a complete page;
generating a portion of a page;
generating page metadata;
generating index data; and
compressing data values.