Processing variable-length fields via formatted record data

1. A method for execution by a record processing and storage system, comprising:

receiving, via a communication network, a record stream from a data source that includes a plurality of formatted record data, wherein each formatted record data of the plurality of formatted record data includes:

a fixed-length batch section that includes a plurality of fixed-length row sections corresponding to a plurality of records of the each formatted record data, wherein each of the plurality of fixed-length row sections includes:

at least one fixed-length data value of a corresponding one of the plurality of records for at least one fixed-length field; and

at least one fixed-length offset value for at least one variable-length field of the corresponding one of the plurality of records;

a variable-length batch section, separate from the fixed-length batch section, that includes a plurality of variable-length row sections corresponding to the plurality of records of the each formatted record data, wherein each of the plurality of variable-length row sections includes:

at least one variable-length data value of a corresponding one of the plurality of records for at least one variable-length field, wherein the fixed-length offset value for the corresponding one of the plurality of records indicates an offset of the at least one variable-length data value in the each formatted record data;

performing parallelized generation of a plurality of pages that each include multiple records of the plurality of formatted record data; and

storing the plurality of pages via a relational database of a page storage system.

2. The method of claim 1 , further comprising:

sending data formatting requirements to the data source;

wherein the plurality of formatted record data is received from the data source in accordance with the data formatting requirements.

3. The method of claim 1 , wherein the plurality of fixed-length row sections are concatenated in sequence, and wherein the variable-length batch section is appended after a final one of the plurality of fixed-length row sections.

4. The method of claim 1 , wherein the plurality of formatted record data is in accordance with row schema data for the plurality of records that indicates a plurality of columns of a corresponding relational database table of the relational database that include the at least one fixed-length field and the at least one variable-length field.

5. The method of claim 4 , wherein the plurality of columns includes a set of multiple fixed-length fields, and wherein the each of the plurality of fixed-length row sections includes a set of multiple fixed-length data values, wherein each fixed-length data value of the set of multiple fixed-length data values corresponds to one of the set of multiple fixed-length fields.

6. The method of claim 5 , wherein a first one of the set of multiple fixed-length fields has a first fixed-length, wherein a second one of the set of multiple fixed-length fields has a second fixed-length different from the first fixed-length, wherein a first fixed-length data value of the each of the plurality of fixed-length row sections has the first fixed-length based on corresponding to the first one of the set of multiple fixed-length fields, and wherein a second fixed-length data value of the each of the plurality of fixed-length row sections has the second fixed-length based on corresponding to the second one of the set of multiple fixed-length fields.

7. The method of claim 4 , wherein the plurality of columns includes a set of multiple variable-length fields, and wherein the each of the plurality of fixed-length row sections includes a set of multiple fixed-length offset values for the set of multiple variable-length field of the corresponding one of the plurality of records, wherein each fixed-length data value of the set of multiple fixed-length offset values for corresponds to one of the set of multiple variable-length fields.

8. The method of claim 1 , further comprising:

executing a read of a variable-length data value of a variable-length field of one of the plurality of records in one of the plurality of pages by:

determining a fixed-length batch section offset value to access the fixed-length offset value for the variable-length field in one of the plurality of fixed-length row sections corresponding to the one of the plurality of records;

determining a variable-length batch section offset value to access the variable-length data value in the variable-length batch section by performing a first read of the one of the plurality of pages based on the fixed-length offset value; and

reading the variable-length data value by performing a second read of the one of the plurality of pages based on the variable-length batch section offset value.

9. The method of claim 8 , wherein determining the fixed-length batch section offset value to access the fixed-length offset value for the variable-length field in one of the plurality of fixed-length row sections corresponding to the one of the plurality of records is based on at least one of:

a fixed-length row section size of the each of the plurality of fixed-length row sections;

an ordering of the plurality of records in the fixed-length batch section;

an ordering of the at least one fixed-length field and the at least one fixed-length offset value in the each of the plurality of fixed-length row sections; or

fixed-lengths of the at least one fixed-length field and the at least one fixed-length offset value.

10. The method of claim 8 , wherein determining the variable-length batch section offset value to access the variable-length data value in the variable-length batch section by performing the first read of the one of the plurality of pages is further based on applying another offset value corresponding to a start of the variable-length batch section.

11. The method of claim 1 , wherein the offset of the at least one variable-length data value in the each formatted record data is a bit offset indicating a location of the at least one variable-length data value in one of the plurality of pages.

12. The method of claim 1 , wherein the plurality of formatted record data is in accordance with a row-based format, further comprising:

generating a plurality of segments from the plurality of pages that include the plurality of records in a column-based format based on; and

storing the plurality of segments via a segment storage system.

13. The method of claim 12 , wherein the plurality of segments are generated based on segment generation determination data indicating segments be generated.

14. A record processing and storage system comprises:

at least one processor; and

a memory that stores operational instructions that, when executed by the at least one processor, cause the record processing and storage system to:

receive, via a communication network, a record stream from a data source that includes a plurality of formatted record data, wherein each formatted record data of the plurality of formatted record data includes:

a fixed-length batch section that includes a plurality of fixed-length row sections corresponding to a plurality of records of the each formatted record data, wherein each of the plurality of fixed-length row sections includes:

at least one fixed-length data value of a corresponding one of the plurality of records for at least one fixed-length field; and

at least one fixed-length offset value for at least one variable-length field of the corresponding one of the plurality of records;

a variable-length batch section, separate from the fixed-length batch section, that includes a plurality of variable-length row sections corresponding to the plurality of records of the each formatted record data, wherein each of the plurality of variable-length row sections includes:

at least one variable-length data value of a corresponding one of the plurality of records for at least one variable-length field, wherein the fixed-length offset value for the corresponding one of the plurality of records indicates an offset of the at least one variable-length data value in the each formatted record data;

perform parallelized generation of a plurality of pages that each include multiple records of the plurality of formatted record data; and

store the plurality of pages via a relational database of a page storage system.

15. The record processing and storage system of claim 14 , wherein the operational instructions, when executed by the at least one processor, further cause the record processing and storage system to:

send data formatting requirements to the data source;

wherein the plurality of formatted record data is received from the data source in accordance with the data formatting requirements.

16. The record processing and storage system of claim 14 , wherein the plurality of fixed-length row sections are concatenated in sequence, and wherein the variable-length batch section is appended after a final one of the plurality of fixed-length row sections.

17. The record processing and storage system of claim 14 , wherein the plurality of formatted record data is in accordance with row schema data for the plurality of records that indicates a plurality of columns of a corresponding relational database table of the relational database that include the at least one fixed-length field and the at least one variable-length field.

18. The record processing and storage system of claim 14 , wherein the operational instructions, when executed by the at least one processor, further cause the record processing and storage system to:

execute a read of a variable-length data value of a variable-length field of one of the plurality of records in one of the plurality of pages by:

determining a fixed-length batch section offset value to access the fixed-length offset value for the variable-length field in one of the plurality of fixed-length row sections corresponding to the one of the plurality of records;

determining a variable-length batch section offset value to access the variable-length data value in the variable-length batch section by performing a first read of the one of the plurality of pages based on the fixed-length offset value; and

reading the variable-length data value by performing a second read of the one of the plurality of pages based on the variable-length batch section offset value.

19. The record processing and storage system of claim 14 , wherein the plurality of formatted record data is in accordance with a row-based format, wherein the operational instructions, when executed by the at least one processor, further cause the record processing and storage system to:

generate a plurality of segments from the plurality of pages that include the plurality of records in a column-based format based on; and

store the plurality of segments via a segment storage system.

20. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing module that includes a processor and a memory, causes the processing module to:

receive, via a communication network, a record stream from a data source that includes a plurality of formatted record data, wherein each formatted record data of the plurality of formatted record data includes:

a fixed-length batch section that includes a plurality of fixed-length row sections corresponding to a plurality of records of the each formatted record data, wherein each of the plurality of fixed-length row sections includes:

at least one fixed-length data value of a corresponding one of the plurality of records for at least one fixed-length field; and

at least one fixed-length offset value for at least one variable-length field of the corresponding one of the plurality of records;

a variable-length batch section, separate from the fixed-length batch section, that includes a plurality of variable-length row sections corresponding to the plurality of records of the each formatted record data, wherein each of the plurality of variable-length row sections includes:

at least one variable-length data value of a corresponding one of the plurality of records for at least one variable-length field, wherein the fixed-length offset value for the corresponding one of the plurality of records indicates an offset of the at least one variable-length data value in the each formatted record data;

perform parallelized generation of a plurality of pages that each include multiple records of the plurality of formatted record data; and

store the plurality of pages via a relational database of a page storage system.