Systems and methods for modularizing data flows
Systems, devices, and methods for modularizing data flows and for providing stand-alone execution of incomplete data flows are provided based on an improved data warehousing framework. Data flow modularization and code reuse is improved through the use of the disclosed flowlets and allows for consistent and efficient standalone testing of incomplete data flows. The disclosed details enable various refinements and modifications according to system design considerations.
1. A computer-implemented method of creating a modularized data flow logic portion comprising:
creating a flowlet definition for a flowlet by providing one or more data flowlet components, each of the data flowlet components having flowlet internals, and one or more of inputs and outputs;
providing a flowlet reference component configured to link the flowlet definition to one or more external data flows, each of the one or more external data flows comprising a SQL Server Integration Services (SSIS) data flow having one or more of inputs and outputs, thereby configuring the flowlet for incorporation into each of the one or more SSIS data flows;
providing a flowlet metadata mapping component configured to isolate boundaries of the flowlet definition logic from the one or more external SSIS data flows by mapping source and destination flowlet component inputs or outputs to the flowlet reference component, which configures the flowlet for execution and testing as a stand-alone data flow; and
merging the flowlet to the one or more SSIS data flows.
2. The method of claim 1 further comprising providing a package component configured to hold the modularized data flow logic portion.
3. The method of claim 1 , wherein providing one or more data flowlet components includes providing a source flowlet component configured as a functional data source in the data flow logic portion.
4. The method of claim 1 , wherein providing one or more data flowlet components includes providing a destination flowlet component configured as a functional data destination in the data flow logic portion.
5. The method of claim 1 , wherein providing one or more data flowlet components includes providing an operation flowlet component for an unknown operation.
6. A computer readable storage medium comprising computer executable instructions for performing a method comprising:
creating a flowlet definition for a flowlet by providing one or more data flowlet components, each of the data flowlet components having flowlet internals, and one or more of inputs and outputs;
providing a flowlet reference component configured to link the flowlet definition to one or more external data flows, each of the one or more external data flows comprising a SQL Server Integration Services (SSIS) data flow having one or more of inputs and outputs, thereby configures the flowlet for incorporation into each of the one or more SSIS data flows; and
providing a flowlet metadata mapping component configured to isolate boundaries of the flowlet definition logic from the one or more external SSIS data flows by mapping source and destination flowlet component inputs or outputs to the flowlet reference component, which configures the flowlet for execution and testing as a stand-alone data flow; and
merging the flowlet to the one or more SSIS data flows.
7. A computing device comprising a processor and memory storing computer-executable instructions which, when executed by the processor, implement a method comprising:
creating a flowlet definition for a flowlet by providing one or more data flowlet components, each of the data flowlet components having flowlet internals, and one or more of inputs and outputs;
providing a flowlet reference component configured to link the flowlet definition to one or more external data flows, each of the one or more external data flows comprising a SQL Server Integration Services (SSIS) data flow having one or more of inputs and outputs, thereby configures the flowlet for incorporation into each of the one or more SSIS data flows; and
providing a flowlet metadata mapping component configured to isolate boundaries of the flowlet definition logic from the one or more external SSIS data flows by mapping source and destination flowlet component inputs or outputs to the flowlet reference component, which configures the flowlet for execution and testing as a stand-alone data flow; and
merging the flowlet to the one or more SSIS data flows.
8. A computer-implemented method for extracting components of an existing data flow to create a modularized data flow, comprising:
selecting and copying components and associated metadata of the existing data flow to create one or more flowlet definitions, the selection creating one or more of disconnected inputs and outputs in the existing data flow;
generating one or more of flowlet source components and flowlet destination components for associated disconnected inputs or outputs of the existing data flow;
connecting disconnected inputs or outputs of the existing data flow to the generated one or more of flowlet destination and source components;
identifying and storing disconnected referenced input columns in connected flowlet source component output columns;
identifying and copying disconnected output columns in connected flowlet destination component output columns; and
representing the selected components with a single flowlet reference component, wherein the generated flowlet source components are represented as separate inputs on the flowlet reference component and the generated flowlet destinations components are represented as separate outputs on the flowlet reference component.
9. The method of claim 8 , further comprising mapping one or more of flowlet reference component inputs and outputs with the one or more of the disconnected inputs or outputs of the existing data flow.
10. The method of claim 9 , further comprising enumerating on the flowlet reference component one or more of connection managers and variables referenced in the selected components of the existing data flow to map real entities in the existing data flow to the one or more flowlet definitions.
11. A computer readable storage medium comprising computer executable instructions for performing a method comprising:
selecting and copying components and associated metadata of the existing data flow to create one or more flowlet definitions, the selection creating one or more of disconnected inputs and outputs in the existing data flow;
generating one or more of flowlet source components and flowlet destination components for associated disconnected inputs or outputs of the existing data flow;
connecting disconnected inputs or outputs of the existing data flow to the generated one or more of flowlet destination and source components;
identifying and storing disconnected referenced input columns in connected flowlet source component output columns;
identifying and copying disconnected output columns in connected flowlet destination component output columns; and
representing the selected components with a single flowlet reference component, wherein the generated flowlet source components are represented as separate inputs on the flowlet reference component and the generated flowlet destinations components are represented as separate outputs on the flowlet reference component.
12. A computing device comprising a processor and memory storing computer-executable instructions which, when executed by the processor, implement a method comprising:
selecting and copying components and associated metadata of the existing data flow to create one or more flowlet definitions, the selection creating one or more of disconnected inputs and outputs in the existing data flow;
generating one or more of flowlet source components and flowlet destination components for associated disconnected inputs or outputs of the existing data flow;
connecting disconnected inputs or outputs of the existing data flow to the generated one or more of flowlet destination and source components;
identifying and storing disconnected referenced input columns in connected flowlet source component output columns;
identifying and copying disconnected output columns in connected flowlet destination component output columns; and
representing the selected components with a single flowlet reference component, wherein the generated flowlet source components are represented as separate inputs on the flowlet reference component and the generated flowlet destinations components are represented as separate outputs on the flowlet reference component.