Data integration system with programmatic source and target interfaces
In one embodiment, a system is provided for executing bulk data transfers between persistent data stores. Programmatic source interfaces are each associated with a source store, defined according to a source interface specification, and exposed during a transfer to enable extraction from its source store of data entities for loading into any selected target stores during the transfer. Programmatic target interfaces are each associated with a target store, defined according to a common target interface specification, and exposed during a transfer to enable loading into its target store of data entities extracted from any selected source stores during the transfer. Each programmatic interface: (1) provides to its store an abstraction of transfer operations such that custom code need not be developed in connection with its store to enable transfers between its store and any other particular stores; and (2) isolates specific details associated with its store such that custom code need not be developed in connection with the transfer operations to enable transfers between its store and any other particular stores.
1. A computer-implemented system for providing bulk data transfers between one or more data stores, comprising:
a data integration server coupled with the one or more data stores, the data integration server comprising:
a plurality of programmatic source interfaces coupled with one or more source data stores, wherein the plurality of programmatic source interfaces are defined according to a source interface specification and are exposed during the bulk data transfer, one or more data entities are extracted from the one or more source data stores; and
a plurality of programmatic target interfaces coupled with one or more target data stores, wherein the plurality of programmatic target interfaces are defined according to a target interface specification and are exposed during the bulk data transfer, one or more of the extracted data entities are loaded into the one or more target data stores;
one or more relational interfaces used as alternatives to the plurality of programmatic source interfaces and the plurality of programmatic target interfaces, each of the one or more relational interfaces coupled with a corresponding relational data store and exposed within the data integration server during the bulk data transfer, the data integration server reads the one or more data entities directly from and writes the one or more data entities directly to the one or more relational data stores during the bulk data transfer when the data integration server determines whether to use the one or more relational interfaces or the plurality of programmatic source interfaces and the plurality of programmatic target interfaces based on file processing time and one or more performance requirements; and
one or more transformation interfaces that are exposed within the data integration server during the bulk data transfer, the one or more transformation interfaces uncoupling transformation logic from one or more of the plurality of programmatic source interfaces and one or more of the plurality of programmatic target interfaces.
2. The system of claim 1 , wherein the plurality of programmatic source interfaces and the plurality of programmatic target interfaces comprise JAVA interfaces.
3. The system of claim 1 , wherein the plurality of programmatic source interfaces and the plurality of programmatic target interfaces are exposed as an industry standard interface supporting the bulk data transfers according to an industry standard protocol.
4. The system of claim 3 , wherein the data integration server is configured to:
receive a request from a client indicating that the client is extracting data from the one or more source data stores or loading data into the one or more target data stores in accordance with the industry standard protocol;
create the plurality of programmatic source interfaces to enable extraction of the data from the one or more source data stores;
create the plurality of programmatic target interface to enable loading of the data into the one or more target data stores;
for data extraction, as the plurality of programmatic source interfaces produce the data extracted from the one or more source data stores, send the data to the client in accordance with the industry standard protocol; and
for data loading, as the data arrives from the client in accordance with the industry standard protocol, send the data to the plurality of programmatic target interfaces for loading into the one or more target data stores.
5. The system of claim 1 , wherein loading the one or more data entities comprises inserting, updating, or deleting the one or more data entities.
6. The system of claim 1 , wherein the data integration server is configured to:
define one or more resources within each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces which represent the one or more data entities within the one or more data stores; and
in response to a request to execute the bulk data transfer involving the one or more resources within the one or more data stores, create each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces within which at least one of the resources is defined.
7. The system of claim 6 , wherein at least one of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces persists, once created:
when the programmatic source interface, for the entirety of the bulk data transfer before being released; and
when the programmatic target interface, for a single step of the bulk data transfer before being released.
8. The system of claim 6 , further comprising one or more session interfaces and wherein:
the plurality of programmatic source interfaces and the plurality of programmatic target interfaces are defined within each of the one or more session interfaces;
each of the one or more session interfaces isolates from the defined programmatic source interface and programmatic target interface details associated with export and import of resources involved in the bulk data transfer; and
the data integration server is further configured to, in connection with creating the plurality of programmatic source interfaces and the plurality of programmatic target interfaces, create each of the one or more session interfaces within which at least one of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces is defined.
9. The system of claim 8 , wherein at least one of the one or more session interfaces persists, once created, either for the entirety of the bulk data transfer or for the entirety of multiple bulk data transfers.
10. The system of claim 1 , wherein the data integration server is configured to:
allow each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces to produce or consume the one or more data entities in a desired format;
convert the one or more data entities produced in a first format to the programmatic source interface to a second format to the programmatic target interface when the first and second formats are different.
11. The system of claim 1 , wherein each of the one or more relational interfaces comprises:
an interface schema file providing a database-neutral description of a physical database schema of the one or more relational data stores; and
an interface mapping file providing a logical-to-physical mapping for all of the one or more data entities defined for the one or more relational interfaces to enable the data integration server to execute the bulk data transfers between one or more of the relational data stores having different physical database schema.
12. A computer-implemented method of executing a bulk data transfer between one or more data stores, comprising:
providing, by a server, a plurality of programmatic source interfaces coupled with one or more source data stores, wherein the plurality of programmatic source interfaces are defined according to a common programmatic source interface specification and are exposed during the bulk data transfer;
extracting, by the server, one or more data entities from the one or more source data stores for loading into one or more selected target data stores;
providing, by the server, a plurality of programmatic target interfaces coupled with one or more target data stores, wherein the plurality of programmatic target interfaces are defined according to a common programmatic target interface specification and are exposed during the bulk data transfer;
providing, by the server, one or more relational interfaces used as alternatives to the plurality of programmatic source interfaces and the plurality of programmatic target interfaces, each of the one or more relational interfaces coupled with a corresponding relational data store and exposed within the server during the bulk data transfer, the server reads the one or more data entities directly from and writes the one or more data entities directly to the one or more relational data stores during the bulk data transfer when the server determines whether to use the one or more relational interfaces or the plurality of programmatic source interfaces and the plurality of programmatic target interfaces based on file processing time and one or more performance requirements;
loading, by the server, the one or more data entities into the one or more target data stores from the one or more source data stores; and
providing, by the server, one or more transformation interfaces that are exposed within the server during the bulk data transfer, the one or more transformation interfaces uncoupling transformation logic from one or more of the plurality of programmatic source interfaces and one or more of the plurality of programmatic target interfaces.
13. The method of claim 12 , wherein the plurality of programmatic source interfaces and the plurality of programmatic target interfaces comprise JAVA interfaces.
14. The method of claim 12 , wherein the plurality of programmatic interfaces are exposed as an industry standard interface supporting the bulk data transfers according to an industry standard protocol.
15. The method of claim 14 , further comprising:
receiving a request from a client indicating that the client is extracting data from the one or more source data stores or loading data into the one or more target data stores in accordance with the industry standard protocol;
creating the plurality of programmatic source interfaces to enable extraction of the data from the one or more source data stores;
creating the plurality of programmatic target interface to enable loading of the data into the one or more target data stores;
for data extraction, as the plurality of programmatic source interface produces the data extracted from the one or more source data stores, sending the data to the client in accordance with the industry standard protocol; and
for data loading, as the data arrives from the client in accordance with the industry standard protocol, sending the data to the plurality of programmatic target interfaces for loading into the one or more target data stores.
16. The method of claim 12 , wherein loading the one or more data entities comprises inserting, updating, or deleting the one or more data entities.
17. The method of claim 12 , further comprising:
defining one or more resources within each of the plurality of programmatic interfaces which represent the one or more data entities within the one or more data stores; and
in response to a request to execute the bulk data transfer involving the one or more resources within the one or more data stores, creating each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces within which at least one of the resources is defined.
18. The method of claim 17 , wherein at least one of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces persists, once created:
when the programmatic source interface, for the entirety of the bulk data transfer before being released; and
when the programmatic target interface, for a single step of the bulk data transfer before being released.
19. The method of claim 17 , further comprising providing one or more session interfaces, wherein:
the plurality of programmatic source interfaces and the plurality of programmatic target interfaces are defined within each of the one or more session interfaces;
each of the one or more session interfaces isolates from the one or more defined programmatic source and target interfaces details associated with export and import of resources involved in the bulk data transfer; and
in connection with creating the plurality of programmatic source interfaces and the plurality of programmatic target interfaces, each of the one or more session interfaces is created within which at least one of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces is defined.
20. The method of claim 19 , wherein at least one of the one or more session interfaces persists, once created, either for the entirety of the bulk data transfer or for the entirety of multiple bulk data transfers.
21. The method of claim 12 , further comprising:
allowing each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces to produce or consume the one or more data entities in a desired format;
converting the one or more data entities produced in a first format to the programmatic source interface to a second format to the programmatic target interface when the first and second formats are different.
22. The method of claim 12 , wherein each of the one or more relational interfaces comprises:
an interface schema file providing a database-neutral description of a physical database schema of the one or more relational data stores; and
an interface mapping file providing a logical-to-physical mapping for all of the one or more data entities defined for the one or more relational interfaces to enable the server to execute the bulk data transfers between one or more of the relational data stores having different physical database schema.
23. A non-transitory computer-readable medium embodied with software for executing a bulk data transfer between one or more data stores, the software when executed using one or more computers is configured to:
provide a plurality of programmatic source interfaces coupled with one or more source data stores, wherein the plurality of programmatic source interfaces are defined according to a common programmatic source interface specification and are exposed during the bulk data transfer;
extract one or more data entities from the one or more source data stores for loading into one or more selected target data stores;
provide a plurality of programmatic target interfaces coupled with one or more target data stores, wherein the plurality of programmatic target interfaces are defined according to a common programmatic target interface specification and are exposed during the bulk data transfer;
provide one or more relational interfaces used as alternatives to the plurality of programmatic source interfaces and the plurality of programmatic target interfaces, each of the one or more relational interfaces coupled with a corresponding relational data store and exposed within a data integration server during the bulk data transfer, the data integration server reads the one or more data entities directly from and writes the one or more data entities directly to the one or more relational data stores during the bulk data transfer when the data integration server determines whether to use the one or more relational interfaces or the plurality of programmatic source interfaces and the plurality of programmatic target interfaces based on file processing time and one or more performance requirements;
load the one or more data entities into the one or more target data stores from the one or more source data stores; and
provide one or more transformation interfaces that are exposed within the data integration server during the bulk data transfer, the one or more transformation interfaces uncoupling transformation logic from one or more of the plurality of programmatic source interfaces and one or more of the plurality of programmatic target interfaces.
24. The computer-readable medium of claim 23 , wherein the plurality of programmatic source interfaces and the plurality of programmatic target interfaces comprise JAVA interfaces.
25. The computer-readable medium of claim 23 , wherein the plurality of programmatic source interfaces and the plurality of programmatic target interfaces are exposed as an industry standard interface supporting the bulk data transfers according to an industry standard protocol.
26. The computer-readable medium of claim 25 , wherein the software is further configured to:
receive a request from a client indicating that the client is extracting data from the one or more source data stores or loading data into the one or more target data stores in accordance with the industry standard protocol;
create the plurality of programmatic source interfaces to enable extraction of the data from the one or more source data stores;
create the plurality of programmatic target interfaces to enable loading of the data into the one or more target data stores;
for data extraction, as the plurality of programmatic source interface produces the data extracted from the one or more source data stores, send the data to the client in accordance with the industry standard protocol; and
for data loading, as the data arrives from the client in accordance with the industry standard protocol, send the data to the plurality of programmatic target interfaces for loading into the one or more target data stores.
27. The computer-readable medium of claim 23 , wherein loading the one or more data entities comprises inserting, updating, or deleting the one or more data entities.
28. The computer-readable medium of claim 23 , wherein the software is further configured to:
define one or more resources within each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces which represent the one or more data entities within the one or more data stores; and
in response to a request to execute the bulk data transfer involving the one or more resources within the one or more data stores, create each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces within which at least one of the resources is defined.
29. The computer-readable medium of claim 28 , wherein at least one of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces persists, once created:
when the programmatic source interface, for the entirety of the bulk data transfer before being released; and
when the programmatic target interface, for a single step of the bulk data transfer before being released.
30. The computer-readable medium of claim 28 , wherein the software is further configured to provide one or more session interfaces, wherein:
the plurality of programmatic source interfaces and the plurality of programmatic target interfaces are defined within each of the one or more session interfaces;
each of the one or more session interfaces isolates from the plurality of programmatic source interfaces and the plurality of programmatic target interfaces details associated with export and import of resources involved in the bulk data transfer; and
in connection with creating the plurality of programmatic source interfaces and the plurality of programmatic target interfaces, create each of the one or more session interfaces within which at least one of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces is defined.
31. The computer-readable medium of claim 30 , wherein at least one of the one or more session interface persists, once created, either for the entirety of the bulk data transfer or for the entirety of multiple bulk data transfers.
32. The computer-readable medium of claim 23 , wherein the software is further configured to:
allow each of the plurality of programmatic source interfaces and the plurality of programmatic target interfaces to produce or consume the one or more data entities in a desired format;
convert the one or more data entities produced in a first format to the programmatic source interface to a second format to the programmatic target interface when the first and second formats are different.
33. The computer-readable medium of claim 23 , wherein each of the one or more relational interfaces comprises:
an interface schema file providing a database-neutral description of a physical database schema of the one or more relational data stores; and
an interface mapping file providing a logical-to-physical mapping for all of the one or more data entities defined for the one or more relational interfaces to enable the data integration server to execute the bulk data transfers between one or more of the relational data stores having different physical database schema.