Nested array batch processing
The present application discloses a method, system, and computer system for processing data. The method includes obtaining a query plan for processing input data in response to a query, obtaining the input data, selecting a batch of the input data, creating a metadata structure for the batch, allocating one or more contiguous parts of a memory for processing the batch, processing the batch in accordance with the metadata structure to generate resulting data, and storing each array of the resulting data for the batch in one of the one or more contiguous parts of the memory.
1. A system, comprising:
a memory;
one or more processors configured to:
obtain a query plan for processing input data in response to a query;
obtain the input data;
select a batch of the input data, wherein the batch of the input data comprises a column of a plurality of rows, wherein the column corresponds to an array type such that one or more rows of the plurality of rows comprises an array in a column position for the row;
create a metadata structure for the batch, wherein the metadata structure includes a mapping of child data corresponding to a particular row being stored in a contiguous memory subset of one or more contiguous parts of memory;
allocate the one or more contiguous parts of the memory for processing the batch;
process the batch of the input data in accordance with the metadata structure to generate resulting data; and
store each array of the resulting data for the batch in one of the one or more contiguous parts of the memory.
2. The system of claim 1 , wherein:
an element of the array type corresponds to an integer, a string, a struct, a map, a float, a decimal, or an array class type;
the metadata structure corresponds to a pointer from the batch of the input data to the resulting data; and
the resulting data for each row is stored sequentially in the contiguous part of the memory.
3. The system of claim 2 , wherein the metadata structure corresponding to the resulting data comprises a<child_idx, offset, length> format.
4. The system of claim 2 , wherein the metadata structure corresponding to the resulting data comprises an<offset, length> format.
5. The system of claim 1 , wherein the child data for each element of an array in the particular row are sequentially stored in the contiguous memory subset.
6. The system of claim 5 , wherein if the child data for a particular element of the array comprises a child array, then the child array elements are sequentially stored in the contiguous memory subset.
7. The system of claim 6 , wherein the input data is stored in the one or more contiguous parts of the memory in an ordered manner based on an original configuration of rows of the input data, arrays comprised in the input rows, and child arrays comprised in the arrays for the input rows.
8. The system of claim 1 , wherein the child data for each of the plurality of rows are stored sequentially in different contiguous memory subsets.
9. The system of claim 1 , wherein the input data is stored in the one or more contiguous parts of the memory in an ordered manner based on an original configuration of rows of the input data, arrays comprised in the input rows, and child arrays comprised in the arrays for the input rows.
10. The system of claim 1 , wherein the one or more processors are configured to:
determine a table of responsive data that is responsive to the query based at least in part on:
walking the one or more contiguous parts of the memory to obtain child data; and
generating the table based at least in part the child data.
11. The system of claim 1 , wherein the one or more processors are further configured to:
process the resulting data based at least in part on the query plan to obtain processed resulting data; and
obtain query responsive data based at least in part on transforming the processed resulting data, wherein the transforming of the processed resulting data is based at least in part on the metadata structure.
12. The system of claim 1 , wherein the one or more processors are further configured to:
transform the resulting data to obtain output data, the resulting data being transformed from the metadata structure to an original structure of the input data.
13. The system of claim 1 , wherein the one or more processors are further configured to:
process the resulting data based at least in part on the query plan to obtain processed resulting data; and
obtain query responsive data based at least in part on transforming the processed resulting data, wherein the transforming of the processed resulting data is based at least in part on a schema for the input data.
14. The system of claim 1 , wherein:
the one or more processors are further configured to:
determine a set of batches for the input data;
the batch of input data is selected from among the set of batches; and
the set of batches are determined based at least in part on a batching policy.
15. The system of claim 14 , wherein the set of batches are determined based at least in part on a predefined batching policy.
16. The system of claim 15 , wherein the predefined batching policy indicates that batches are to be determined based on a number of rows or a size of data.
17. A method, comprising:
obtaining a query plan for processing input data in response to a query;
obtaining the input data;
selecting a batch of the input data, wherein the batch of the input data comprises a column of a plurality of rows, wherein the column corresponds to an array type such that one or more rows of the plurality of rows comprises an array in a column position for the row;
creating a metadata structure for the batch, wherein the metadata structure includes a mapping of child data corresponding to a particular row being stored in a contiguous memory subset of one or more contiguous part of memory;
allocating the one or more contiguous parts of a memory for processing the batch;
processing the batch of the input data in accordance with the metadata structure to generate resulting data; and
storing each array of the resulting data for the batch in one of the one or more contiguous parts of the memory.
18. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
obtaining a query plan for processing input data in response to a query;
obtaining the input data;
selecting a batch of the input data, wherein the batch of the input data comprises a column of plurality of rows, wherein the column corresponds to an array type such that one or more rows of the plurality of rows comprises an array in a column position for the row;
creating a metadata structure for the batch, wherein the metadata structure includes a mapping of child data corresponding to a particular row being stored in aa contiguous memory subset of one or more contiguous parts of memory;
allocating the one or more contiguous parts of a memory for processing the batch;
processing the batch of the input data in accordance with the metadata structure to generate resulting data; and
storing each array of the resulting data for the batch in one of the one or more contiguous parts of the memory.