System and method for generating partition-based splits in a massively parallel or distributed database environment

1. A method for database split generation in an associated massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying of the plurality of databases, the method comprising:

obtaining by a database table accessor executing on one or more microprocessors, from an associated client application, a query for data in a table of the data warehouse layer, the query comprising query data and user preference data, the query data being representative of a user query and the user preference data being representative of a user preference for a partition-based splits generator;

obtaining by the database table accessor, from the data warehouse layer in accordance with the user query, table data representative of one or more properties of the table, wherein the table data comprises partition data representative of a partition scheme of the table as having a partitioned topology wherein the table is logically divided into one or more partitions or an un-partitioned topology wherein the table is logically undivided;

selecting by the database table accessor the partition-based splits generator in accordance with:

the query data indicating the user preference for the partition-based splits generator;

the partition data indicating the partition scheme of the table as having the partitioned topology; or

the query data indicating the user preference for the partition-based splits generator and the partition data indicating the partition scheme of the table as having the partitioned topology;

pruning portions of the table from the query by analyzing, by the database table accessor, the table data relative to a predicate of the query data to derive a query valid list of partitions;

using the selected partition-based splits generator, generating one or more table splits for each of the query valid list of partitions of the table, the one or more table splits dividing the user query into a plurality of query splits; and

outputting the plurality of query splits to a plurality of associated mappers for execution by the plurality of associated mappers of each of the plurality of query tasks against the table.

2. The method of claim 1 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

obtaining partition size data representative of a partition size of each of the query valid list of partitions of the table;

obtaining split size data representative of a maximum split size of each of the plurality of query splits; and

comparing the split size data with the partition size data and storing a result of the comparing as comparison data; and

generating the one or more table splits for each of the query valid list of partitions of the table in accordance with the comparison data.

3. The method of claim 2 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

determining the partition size is greater than the maximum split size in accordance with the comparison data;

generating a plurality of table splits for each of the query valid list of partitions in accordance with determining the partition size is greater than the maximum split size.

4. The method of claim 3 , wherein the generating the plurality of table splits comprises:

generating a set of table splits having an aggregate split size equal to or less than the size of each of the query valid list of partitions of the table in accordance with the partition size data.

5. The method of claim 4 , wherein the generating the set of table splits comprises:

generating a maximum number of table splits having an aggregate split size equal to or less than the size of each of the query valid list of partitions of the table in accordance with the partition size data.

6. The method of claim 2 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

determining the partition size is less than the maximum split size in accordance with the comparison data;

generating a single table split for a plurality of partitions in accordance with determining the partition size is less than the maximum split size.

7. The method of claim 6 , wherein the generating the single table split comprises:

generating the single table split for a set of partitions having an aggregate partition size equal to or less than the maximum split size of the single table split in accordance with the split size data.

8. The method of claim 7 , wherein the generating the single table split comprises:

assigning the single table split to a maximum number of partitions of the table having an aggregate partition size equal to or less than the maximum split size of the single table split in accordance with the split size data.

9. A system for database split generation in an associated massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying of the plurality of databases, the system comprising:

one or more microprocessors;

a database table accessor running on the one or more microprocessors, wherein the database table accessor operates to perform steps comprising:

obtaining, by the database table accessor from an associated client application, a query for data in a table of the data warehouse layer, the query comprising query data and user preference data, the query data being representative of a user query and the user preference data being representative of a user preference for a partition-based splits generator;

obtaining by the database table accessor, from the data warehouse layer in accordance with the user query, table data representative of one or more properties of the table, wherein the table data comprises partition data representative of a partition scheme of the table as having a partitioned topology wherein the table is logically divided into one or more partitions or an un-partitioned topology wherein the table is logically undivided;

selecting by the database table accessor the partition-based splits generator in accordance with:

the query data indicating the user preference for the partition-based splits generator;

the partition data indicating the partition scheme of the table as having the partitioned topology; or

the query data indicating the user preference for the partition-based splits generator and the partition data indicating the partition scheme of the table as having the partitioned topology;

pruning portions of the table from the query by analyzing, by the database table accessor, the table data relative to a predicate of the query data to derive a query valid list of partitions;

using the selected partition-based splits generator, generating one or more table splits for each of the query valid list of partitions of the table, the one or more table splits dividing the user query into a plurality of query splits; and

outputting the plurality of query splits to a plurality of associated mappers for execution by the plurality of associated mappers of each of the plurality of query tasks against the table.

10. The system of claim 9 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

obtaining partition size data representative of a partition size of each of the query valid list of partitions of the table;

obtaining split size data representative of a maximum split size of each of the plurality of query splits; and

comparing the split size data with the partition size data and storing a result of the comparing as comparison data; and

generating the one or more table splits for each of the query valid list of partitions of the table in accordance with the comparison data.

11. The system of claim 10 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

determining the partition size is greater than the maximum split size in accordance with the comparison data;

generating a plurality of table splits for each of the query valid list of partitions in accordance with determining the partition size is greater than the maximum split size.

12. The system of claim 11 , wherein the generating the plurality of table splits comprises:

generating a set of table splits having an aggregate split size equal to or less than the size of each of the query valid list of partitions of the table in accordance with the partition size data.

13. The system of claim 12 , wherein the generating the set of table splits comprises:

generating a maximum number of table splits having an aggregate split size equal to or less than the size of each of the query valid list of partitions of the table in accordance with the partition size data.

14. The system of claim 10 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

determining the partition size is less than the maximum split size in accordance with the comparison data;

generating a single table split for a plurality of partitions in accordance with determining the partition size is less than the maximum split size.

15. The system of claim 14 , wherein the generating the single table split comprises:

generating the single table split for a set of partitions having an aggregate partition size equal to or less than the maximum split size of the single table split in accordance with the split size data.

16. The system of claim 15 , wherein the generating the single table split comprises:

assigning the single table split to a maximum number of partitions of the table having an aggregate partition size equal to or less than the maximum split size of the single table split in accordance with the split size data.

17. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers of a database table accessor in an associated massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying of the plurality of databases, cause the one or more computers of the database table accessor to perform steps providing partition-based splits of query tasks comprising:

obtaining by a database table accessor executing on one or more microprocessors of the one or more computers, from an associated client application, a query for data in a table of the data warehouse layer, the query comprising query data and user preference data, the query data being representative of a user query and the user preference data being representative of a user preference for a partition-based splits generator;

obtaining by the database table accessor, from the data warehouse layer in accordance with the user query, table data representative of one or more properties of the table, wherein the table data comprises partition data representative of a partition scheme of the table as having a partitioned topology wherein the table is logically divided into one or more partitions or an un-partitioned topology wherein the table is logically undivided;

selecting by the database table accessor the partition-based splits generator in accordance with:

the query data indicating the user preference for the partition-based splits generator;

the partition data indicating the partition scheme of the table as having the partitioned topology; or

the query data indicating the user preference for the partition-based splits generator and the partition data indicating the partition scheme of the table as having the partitioned topology;

pruning portions of the table from the query by analyzing, by the database table accessor, the table data relative to a predicate of the query data to derive a query valid list of partitions;

using the selected partition-based splits generator, generating one or more table splits for each of the query valid list of partitions of the table, the one or more table splits dividing the user query into a plurality of query splits; and

outputting the plurality of query splits to a plurality of associated mappers for execution by the plurality of associated mappers of each of the plurality of query tasks against the table.

18. The non-transitory computer readable storage medium of claim 17 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

obtaining partition size data representative of a partition size of each of the query valid list of partitions of the table;

obtaining split size data representative of a maximum split size of each of the plurality of query splits; and

comparing the split size data with the partition size data and storing a result of the comparing as comparison data; and

generating the one or more table splits for each of the query valid list of partitions of the table in accordance with the comparison data.

19. The non-transitory computer readable storage medium of claim 18 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

determining the partition size is greater than the maximum split size in accordance with the comparison data;

generating a plurality of table splits for each of the query valid list of partitions in accordance with determining the partition size is greater than the maximum split size.

20. The non-transitory computer readable storage medium of claim 18 , wherein generating the one or more table splits for each of the query valid list of partitions of the table comprises:

determining the partition size is less than the maximum split size in accordance with the comparison data;

generating a single table split for a plurality of partitions in accordance with determining the partition size is less than the maximum split size.