Abstract pipeline component connection
View Patent ↗The subject disclosure pertains to systems and methods for connecting pipeline components such as data flow pipeline components. Pipeline components are wrapped in metadata components that include the component as well as abstracted metadata in the form of input and output metadata descriptors. A connector can arbitrate metadata differences between pipeline components based on the metadata descriptors, thereby preventing upstream modifications from invalidating or breaking components downstream. Native or plug-in heuristic and/or related components can provide the functionality necessary to resolve metadata differences.
1. A computer program product for a pipeline system comprising one or more computer-readable storage media having encoded thereon computer executable instructions which, when executed upon one or more computer processors, provide a plurality of computer components, the components comprising:
a plurality of metadata components;
a reference component, the reference component identifying reference data which is used in resolving metadata differences; and
a connector component, the connector component comprising three subcomponents, the three subcomponents consisting of (i) a receiver component, (ii) an execution component, (iii) and a transmission component,
the receiver component acquiring data and metadata from a first metadata component of the plurality of metadata components and providing the acquired data and metadata to the execution component for processing,
the execution component being communicatively coupled to one or more heuristic components, the one or more heuristic components providing functionality for resolving metadata mismatches,
the transmission component receiving arbitrated data from the execution component for transmission to a downstream component,
the connector component analyzing metadata associated with at least two metadata components of the plurality of metadata components, and
the connector component arbitrating metadata differences between the at least two metadata components by manipulation of the metadata associated with a first metadata component based at least in part on the execution component resolving metadata mismatches such that the metadata associated with the first metadata component is compatible with a schema associated with the at least one other metadata component,
the execution component providing arbitrated data to the transmission component, and
the transmission component transmitting the arbitrated data to a downstream metadata component.
2. The computer program product of claim 1 , wherein at least one of the plurality of metadata components comprises at least one input metadata descriptor and at least one output metadata descriptor.
3. The computer program product of claim 1 , wherein the execution component executes functionality provided by at least one of the one or more heuristic components.
4. The computer program product of claim 1 , wherein the receiver component acquires an input descriptor from the downstream metadata component.
5. The computer program product of claim 3 , further comprising an interface component to enable a user to modify a particular heuristic component.
6. The computer program product of claim 5 , further comprising a generation component communicatively coupled to the interface component that facilitates producing a new heuristic component.
7. The computer program product of claim 5 , further comprising one or more interface components to facilitate receipt of non-native or plug-in heuristic components.
8. The computer program product of claim 3 , the at least one of the one or more heuristic components employs a file that contains reference data to facilitate transforming data from a first format or first schema associated with a first domain to a disparate format or disparate schema associated with a disparate domain.
9. The computer program product of claim 3 , wherein the at least one of the one or more heuristic components:
analyzes at least one of data or metadata associated with the first metadata component and the downstream metadata component,
infers one or more types of data respectively associated with the first metadata component and the downstream metadata component based at least in part on the analysis of the at least one of data or metadata, and
automatically determines a mapping between the first metadata component and the downstream metadata component that is utilized to facilitate arbitration of metadata differences between the first metadata component and the downstream metadata component.
10. A computer-implemented method for a pipeline system, the method comprising executing computer-executable instructions upon one or more computer processors, the computer-executable instructions, when executed upon the one or more computer processors, providing a plurality of computer components, the components comprising:
a plurality of metadata components;
a reference component, the reference component identifying reference data which is used in resolving metadata differences; and
a connector component, the connector component comprising three subcomponents, the three subcomponents consisting of (i) a receiver component, (ii) an execution component, (iii) and a transmission component,
the receiver component acquiring data and metadata from a first metadata component of the plurality of metadata components and providing the acquired data and metadata to the execution component for processing,
the execution component being communicatively coupled to one or more heuristic components, the one or more heuristic components providing functionality for resolving metadata mismatches,
the transmission component receiving arbitrated data from the execution component for transmission to a downstream component,
the connector component analyzing metadata associated with at least two metadata components of the plurality of metadata components, and
the connector component arbitrating metadata differences between the at least two metadata components by manipulation of the metadata associated with a first metadata component based at least in part on the execution component resolving metadata mismatches such that the metadata associated with the first metadata component is compatible with a schema associated with the at least one other metadata component,
the execution component providing arbitrated data to the transmission component, and
the transmission component transmitting the arbitrated data to a downstream metadata component.
11. The method of claim 10 , wherein at least one of the plurality of metadata components comprises at least one input metadata descriptor and at least one output metadata descriptor.
12. The method of claim 10 , wherein the execution component executes functionality provided by at least one of the one or more heuristic components.
13. The method of claim 10 , wherein the receiver component acquires an input descriptor from the downstream metadata component.
14. The method of claim 12 , further comprising an interface component to enable a user to modify a particular heuristic component.
15. The method of claim 14 , further comprising a generation component communicatively coupled to the interface component that facilitates producing a new heuristic component.
16. The method of claim 14 , further comprising one or more interface components to facilitate receipt of non-native or plug-in heuristic components.
17. The method of claim 12 , the at least one of the one or more heuristic components employs a file that contains reference data to facilitate transforming data from a first format or first schema associated with a first domain to a disparate format or disparate schema associated with a disparate domain.
18. The method of claim 12 , wherein the at least one of the one or more heuristic components:
analyzes at least one of data or metadata associated with the first metadata component and the downstream metadata component,
infers one or more types of data respectively associated with the first metadata component and the downstream metadata component based at least in part on the analysis of the at least one of data or metadata, and
automatically determines a mapping between the first metadata component and the downstream metadata component that is utilized to facilitate arbitration of metadata differences between the first metadata component and the downstream metadata component.