Method and apparatus with data partitioning and parallel processing for transporting data for data warehousing applications
View Patent ↗A method and apparatus for transporting data for a data warehouse application is described. The data from an operational data store (the source database) is organized in non-overlapping data partitions. Separate execution threads read the data from the operational data store concurrently. This is followed by concurrent transformation of the data in multiple execution threads. Finally, the data is loaded into the target data warehouse concurrently using multiple execution threads. By using multiple execution threads, the data contention is reduced. Thereby the apparatus and method of the present invention achieves increased throughput.
1. A computer implemented method for transporting data in a data warehousing application, comprising the steps of:
specifying at least one source containing data wherein at least some portion of said data is to be transported;
partitioning formerly un-partitioned data from said source containing data so as to form a plurality of non overlapping data portions;
constructing a plurality of pipelines that include transformation components for manipulating data in said data partitions; and
specifying a target for storing data generated by one or more pipelines.
2. The computer implemented method of claim 1 wherein partitioning formerly un-partitioned data further comprises the steps of:
receiving user input that indicates desired data partitioning; and
partitioning data from said source containing data so as to form a plurality of data partitions conforming to said user input.
3. The computer implemented method of claim 2 wherein said data is partitioned by dividing said data evenly into a user-selected number of data partitions.
4. The computer implemented method of claim 1 wherein constructing a plurality of pipelines further comprises the steps of:
constructing a plurality of transformation components for manipulating data in said data partitions; and
coupling the transformation components to form a plurality of pipelines.
5. The computer implemented method of claim 4 wherein coupling of transformation components to form a plurality of pipelines is performed such that at least some of said plurality of pipelines operate using data from a single data partition.
6. The computer implemented method of claim 4 wherein coupling of transformation components to form a plurality of pipelines allows for multiple pipelines to access each of said data partitions.
7. The computer implemented method of claim 1 further comprising the step of:
executing a plurality of tasks in parallel through said plurality of pipelines.
8. The computer-implemented method of claim 6 wherein said data is partitioned such that there is affinity for data within each data partition.
9. The computer-implemented method of claim 6 wherein at least some of said pipelines are independent execution threads.
10. A computer readable medium having stored therein instructions for causing a computer to implement a method for transporting data in a data warehousing application, said method comprising the steps of:
specifying at least one source containing data wherein at least some portion of said data is to be transported;
partitioning formerly un-partitioned data from said source containing data so as to form a plurality of non overlapping data portions;
constructing a plurality of pipelines that include transformation components for manipulating data in said data partitions; and
specifying a target for storing data generated by one or more pipelines.
11. A computer readable medium as described in claim 10 wherein partitioning formerly un-partitioned data further comprises:
receiving user input that indicates desired data partitioning; and
partitioning data from said source containing data so as to form a plurality of data partitions conforming to said user input.
12. A computer readable medium as recited in claim 10 wherein said data is partitioned by dividing said data evenly into a user-selected number of data partitions.
13. A computer readable medium as recited in claim 10 wherein constructing a plurality of pipelines further comprises the steps of:
constructing a plurality of transformation components for manipulating data in said data partitions; and
coupling the transformation components to form a plurality of pipelines.
14. A computer readable medium as recited in claim 13 wherein data is partitioned such that there is affinity for data contained within each data partition.
15. A computer readable medium as recited in claim 13 wherein coupling the transformation components to form a plurality of pipelines allows for coupling of said transformation components such that multiple pipelines can access each of said data partitions.
16. A computer readable medium as recited in claim 10 wherein said method further comprises the step of:
executing a plurality of tasks in parallel through said plurality of pipelines.
17. A computer readable medium as recited in claim 13 wherein said pipelines are formed so as to minimize data sharing between pipelines.
18. The computer-readable medium of claim 13 , wherein said transformation components include a source transformation component, a target transformation component, an aggregation transformation component, a rank transformation component, and a joiner transformation component that stage data.
19. The computer-readable medium of claim 13 , wherein said transformation components include an expression transformation component, a filter transformation component, an update strategy transformation component, a sequence transformation component, a lookup transformation component, a stored procedure transformation component, an external procedure transformation component, and a normalizer transformation component for streaming data.
20. A method for transporting data in a data warehousing application comprising:
partitioning formerly un-partitioned data from said source containing data to form a plurality of overlapping data partitions;
storing at least some of said partitioned data;
constructing a plurality of pipelines that include transformation components for manipulating data in said data partitions;
coupling said transformation components to form a plurality of parallel pipelines; and
executing a plurality of tasks in parallel through said plurality of pipelines; and
storing said data generated by one or more of said pipelines in a target database.
21. The method of claim 20 wherein said transformation components are coupled such that said plurality of pipelines correspond to said plurality of data partitions, thereby reducing the amount of sharing of data between individual pipelines.
22. The computer implemented method of claim 21 wherein data affinity factors are used in constructing said plurality of transformation components.