Systems, apparatus, and methods for data integration optimization
Systems, methods, and techniques for optimizing a plurality of data integration tasks within a data integration collection by identifying, as a sub-set of the plurality of data integration tasks, a plurality of point-to-point data integration tasks defining a data integration transformation plan to include: generating one or more publication data integration tasks comprising publishing from each respective data source of the plurality of point-to-point data integration tasks to generate a single publication topic; and generating one or more subscription data integration tasks causing each respective target of the plurality of point-to-point data integration tasks to subscribe to the single publication topic; and generating a set of optimization instructions configured to cause the at least one computer to implement the data integration transformation plan; and executing the set of optimization instructions to generate the one or more publication data integration tasks and the one or more subscription data integration tasks.
1. A method executed by one or more computing devices for optimizing a plurality of data integration tasks, the method comprising:
identifying, by at least one of the one or more computing devices, as a sub-set of the plurality of data integration tasks, a plurality of point-to-point data integration tasks each corresponding to a respective data source and a respective target;
generating, by at least one of the one or more computing devices, one or more publication data integration tasks by publishing from each respective data source of the plurality of point-to-point data integration tasks to a single publication topic in place of each respective target;
generating, by at least one of the one or more computing devices, one or more subscription data integration tasks by causing each respective target of the plurality of point-to-point data integration tasks to subscribe to the single publication topic;
optimizing, by at least one of the one or more computing devices, the plurality of data integration tasks by replacing the plurality of point-to-point data integration tasks with the one or more publication data integration tasks and the one or more subscription data integration tasks, such that the one or more publication data integration tasks and the one or more subscription data integration tasks are executed in place of the plurality of point-to-point data integration tasks when data integration is performed; and
executing, by at least one of the one or more computing devices, the plurality of data integration tasks including the one or more publication data integration tasks and the one or more subscription integration tasks.
2. The method of claim 1 , further comprising:
generating, by at least one of the one or more computing devices, a source integration map comprising at least one first key value pairs, the at least one first key value pair comprising a first key generated by applying a hash function to at least one respective data source and at least one first value describing one or more integration tasks associated with the at least one respective data source.
3. The method of claim 2 , further comprising:
generating, by at least one of the one or more computing devices, a target integration comprising at least one second key value pair, the at least one second key value pair comprising a second key generated by applying a hash function to at least one respective data target and at least one first value describing one or more integration tasks associated with the at least one respective data target.
4. The method of claim 3 , further comprising:
generating, by at least one of the one or more computing devices, the single publication topic based on the at least one first key value pair or the at least one second key value pair.
5. The method of claim 1 , wherein the single publication topic is deleted after the one or more subscription data integration tasks execute.
6. The method of claim 1 , further comprising:
deleting, by at least one of the one or more computing devices, the plurality of point-to-point data integration tasks.
7. An apparatus for optimizing a plurality of data integration tasks, the apparatus comprising:
one or more processors; and
one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
identify as a sub-set of the plurality of data integration tasks, a plurality of point-to-point data integration tasks each corresponding to a respective data source and a respective target;
generate one or more publication data integration tasks by publishing from each respective data source of the plurality of point-to-point data integration tasks to a single publication topic in place of each respective target;
generate one or more subscription data integration tasks by causing each respective target of the plurality of point-to-point data integration tasks to subscribe to the single publication topic;
optimize the plurality of data integration tasks by replacing the plurality of point-to-point data integration tasks with the one or more publication data integration tasks and the one or more subscription data integration tasks, such that the one or more publication data integration tasks and the one or more subscription data integration tasks are executed in place of the plurality of point-to-point data integration tasks when data integration is performed; and
execute the plurality of data integration tasks including the one or more publication data integration tasks and the one or more subscription integration tasks.
8. The apparatus of claim 7 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
generate a source integration map comprising at least one first key value pairs, the at least one first key value pair comprising a first key generated by applying a hash function to at least one respective data source and at least one first value describing one or more integration tasks associated with the at least one respective data source.
9. The apparatus of claim 8 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
generate a target integration comprising at least one second key value pair, the at least one second key value pair comprising a second key generated by applying a hash function to at least one respective data target and at least one first value describing one or more integration tasks associated with the at least one respective data target.
10. The apparatus of claim 9 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
generate the single publication topic based on the at least one first key value pair or the at least one second key value pair.
11. The apparatus of claim 7 , wherein the single publication topic is deleted after the one or more subscription data integration tasks execute.
12. The apparatus of claim 7 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
delete the plurality of point-to-point data integration tasks.
13. At least one non-transitory computer-readable medium storing computer-readable instructions for optimizing a plurality of data integration tasks that, when executed by one or more computing devices, cause at least one of the one or more computing devices to:
identify as a sub-set of the plurality of data integration tasks, a plurality of point-to-point data integration tasks each corresponding to a respective data source and a respective target;
generate one or more publication data integration tasks by publishing from each respective data source of the plurality of point-to-point data integration tasks to a single publication topic in place of each respective target;
generate one or more subscription data integration tasks by causing each respective target of the plurality of point-to-point data integration tasks to subscribe to the single publication topic;
optimize the plurality of data integration tasks by replacing the plurality of point-to-point data integration tasks with the one or more publication data integration tasks and the one or more subscription data integration tasks, such that the one or more publication data integration tasks and the one or more subscription data integration tasks are executed in place of the plurality of point-to-point data integration tasks when data integration is performed; and
execute the plurality of data integration tasks including the one or more publication data integration tasks and the one or more subscription integration tasks.
14. The at least one non-transitory computer-readable medium of claim 13 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:
generate a source integration map comprising at least one first key value pairs, the at least one first key value pair comprising a first key generated by applying a hash function to at least one respective data source and at least one first value describing one or more integration tasks associated with the at least one respective data source.
15. The at least one non-transitory computer-readable medium of claim 14 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:
generate a target integration comprising at least one second key value pair, the at least one second key value pair comprising a second key generated by applying a hash function to at least one respective data target and at least one first value describing one or more integration tasks associated with the at least one respective data target.
16. The at least one non-transitory computer-readable medium of claim 15 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:
generate the single publication topic based on the at least one first key value pair or the at least one second key value pair.
17. The at least one non-transitory computer-readable medium of claim 13 , wherein the single publication topic is deleted after the one or more subscription data integration tasks execute.
18. The at least one non-transitory computer-readable medium of claim 13 , further storing computer-readable instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:
delete the plurality of point-to-point data integration tasks.