System and method for dynamic database split generation in a massively parallel or distributed database environment
A system and method is described for database split generation in a massively parallel or other distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying functionality. A database table accessor of the system obtains, from an associated client application, a query for data in a table of the data warehouse layer, wherein the query includes a user preference. The system obtains table data representative of properties of the table, and determines a splits generator in accordance with one or more of the user preference or the properties of the table. The system generates, by the selected splits generator, table splits dividing the user query into a plurality of query splits, and outputs the plurality of query splits to an associated plurality of mappers for execution by the associated plurality of mappers of each of the plurality of query splits against the table.
1. A method for database split generation in a database access environment including a plurality of databases and a data warehouse layer executing on one or more microprocessors, the method comprising:
receiving by a database table accessor executing on the one or more microprocessors, from a client application, a query for data in a table of the data warehouse layer;
obtaining a metadata representative of the table containing the data;
determining a splits generator based at least partly on the obtained metadata;
generating, by the splits generator, a plurality of query splits, wherein each of the plurality of query splits is executable by a corresponding associated mapper of a plurality of associated mappers against the table; and
generating a response to the query based on received results of each of the plurality of query splits executed against the table.
2. The method of claim 1 , wherein the determining the splits generator comprises selecting the splits generator from among a plurality of splitter kinds in accordance with a user preference or one or more properties of the table.
3. The method of claim 2 , wherein the determining the splits generator comprises selecting the splits generator for splitting the table based on one or more of partitions, sizes, row limits, and/or ROWID ranges.
4. The method of claim 2 , wherein the determining the splits generator comprises determining the splits generator exclusively in accordance with:
the user preference, or
the one or more properties of the table.
5. The method of claim 1 , wherein the obtaining the metadata comprises obtaining metadata representative of the one or more properties of the table.
6. The method of claim 1 , further comprising:
executing each of the plurality of query splits against the table using a corresponding associated mapper of the plurality of associated mappers.
7. The method of claim 1 , further comprising:
outputting the plurality of query splits for execution against the table by the corresponding associated mappers of the plurality of mappers;
receiving results of each of the plurality of query splits executed against the table using the corresponding associated mappers of the plurality of mappers; and
generating the response based on the received results of each of the plurality of query splits executed against the table using the corresponding associated mappers of the plurality of mappers.
8. The method of claim 1 , wherein:
obtaining metadata comprises obtaining table data representative of one or more properties of the table from a metadata store, wherein the table data comprises one or more of:
partition data representative of a partition scheme of the table as having a partitioned topology wherein the table is logically divided into one or more partitions or an un-partitioned topology wherein the table is logically undivided; and/or
files, data blocks, and row numbers for the table in accordance with the user query for blocks of data in the table.
9. A system for database split generationin a database access environment including a plurality of databases and a data warehouse layer executing on one or more microprocessors, the system comprising:
one or more microprocessors; and
a database table accessor executing on the one or more microprocessors and being configured to perform a method of database split generation comprising:
receiving by a database table accessor executing on the one or more microprocessors, from a client application, a query for data in a table of the data warehouse layer;
obtaining a metadata representative of the table containing the data;
determining a splits generator based at least partly on the obtained metadata;
generating, by the splits generator, a plurality of query splits, wherein each of the plurality of query splits is executable by a corresponding associated mapper of a plurality of associated mappers against the table; and
generating a response to the query based on received results of each of the plurality of query splits executed against the table.
10. The system of claim 9 , wherein the determining the splits generator comprises selecting the splits generator from among a plurality of splitter kinds in accordance with a user preference or one or more properties of the table.
11. The system of claim 10 , wherein the determining the splits generator comprises selecting the splits generator for splitting the table based on one or more of partitions, sizes, row limits, and/or ROWID ranges.
12. The system of claim 10 , wherein the determining the splits generator comprises determining the splits generator exclusively in accordance with:
the user preference, or
the one or more properties of the table.
13. The system of claim 9 , wherein the obtaining the metadata comprises obtaining metadata representative of the one or more properties of the table.
14. The system of claim 9 , further comprising:
executing each of the plurality of query splits against the table using a corresponding associated mapper of the plurality of associated mappers.
15. The system of claim 9 , further comprising:
outputting the plurality of query splits for execution against the table by the corresponding associated mappers of the plurality of mappers;
receiving results of each of the plurality of query splits executed against the table using the corresponding associated mappers of the plurality of mappers; and
generating the response based on the received results of each of the plurality of query splits executed_against the table using the corresponding associated mappers of the plurality of mappers.
16. The system of claim 9 , wherein:
obtaining metadata comprises obtaining table data representative of one or more properties of the table from a metadata store, wherein the table data comprises one or more of:
partition data representative of a partition scheme of the table as having a partitioned topology wherein the table is logically divided into one or more partitions or an un-partitioned topology wherein the table is logically undivided; and/or
files, data blocks, and row numbers for the table in accordance with the user query for blocks of data in the table.
17. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers, cause the one or more computers to perform a method of database split generation comprising:
receiving_by a database table accessor executing on the one or more microprocessors, from a client application, a query for data in a table of the data warehouse layer;
obtaining a metadata representative of the table containing the data;
determining a splits generator based at least partly on the obtained metadata;
generating, by the splits generator, a plurality of query splits, wherein each of the plurality of query splits is executable by a corresponding associated mapper of a plurality of associated mappers against the table; and
generating a response to the query based on received results of each of the plurality of query splits executed against the table.
18. The non-transitory computer readable storage medium of claim 17 , wherein the determining the splits generator comprises selecting the splits generator from among a plurality of splitter kinds in accordance with a user preference or one or more properties of the table.
19. The non-transitory computer readable storage medium of claim 17 , further comprising:
executing each of the plurality of query splits against the table using a corresponding associated mapper of the plurality of associated mappers.
20. The non-transitory computer readable storage medium of claim 17 , further comprising:
outputting the plurality of query splits for execution against the table by the corresponding associated mappers of the plurality of mappers;
receiving results of each of the plurality of query splits executed against the table using the corresponding associated mappers of the plurality of mappers; and
generating the response based on the received results of each of the plurality of query splits executed against the table using the corresponding associated mappers of the plurality of mappers.