System, method, and apparatus for synchronization among heterogeneous data sources
A method for synchronizing data among heterogeneous data sources is disclosed. The method may comprise obtaining a synchronization task and a corresponding operation classification, determining one or more Reader plugins and Writer plugins based on the corresponding operation classification, segmenting the synchronization task based on the one or more Reader plugins and Writer plugins to generate a Task queue, and using a corresponding number of the Reader plugins and the Writer plugins based on the Task queue to implement the synchronization task. The one or more Reader plugins may be configured to read data from a source and to send to the one or more Writer plugins, and the one or more Writer plugins may be configured to write the data to a destination.
1. A method for synchronizing data among heterogeneous data sources, implemented by a processor, comprising:
obtaining a synchronization task and a corresponding operation classification;
determining one or more Reader plugins and one or more Writer plugins based on the corresponding operation classification, wherein the corresponding operation classification indicates whether the synchronization task is between different types of database systems or between a same type of database system, and wherein the one or more Reader plugins are associated with a first type of data source, the one or more Writer plugins are associated with a second type of data source, and the first type of data source is different from or the same with the second type of data source;
segmenting the synchronization task based on the one or more Reader plugins and Writer plugins to generate a Task queue, the one or more Reader plugins being configured to read data from a source and to send to the one or more Writer plugins, and the one or more Writer plugins being configured to write the data to a destination, wherein tasks in the Task queue are processed in parallel; and
implementing the synchronization task using a corresponding number of the Reader plugins and the Writer plugins based on the Task queue.
2. The method of claim 1 , wherein:
the synchronization task is obtained by a first ClassLoader; and
the corresponding number of the Reader plugins and the Writer plugins based on the Task queue are used by a second ClassLoader.
3. The method of claim 1 , wherein the one or more Reader plugins are configured to send the data to the Writer plugins via a channel between the Reader plugins and the Writer plugins.
4. The method of claim 1 , wherein the one or more Reader plugins and the Writer plugins each have an independent thread.
5. The method of claim 1 , wherein the one or more Reader plugins and the Writer plugins include init( ), prepare( ), post( ), and destroy( ) methods.
6. An apparatus for synchronizing data among heterogeneous data sources comprising:
a memory storing instructions; and
a processor configured to execute the instructions to cause the apparatus to perform:
obtaining a synchronization task and a corresponding operation classification;
determining one or more Reader plugins and one or more Writer plugins based on the corresponding operation classification, wherein the corresponding operation classification indicates whether the synchronization task is between different types of database systems or between a same type of database system, and wherein the one or more Reader plugins are associated with a first type of data source, the one or more Writer plugins are associated with a second type of data source, and the first type of data source is different from or the same with the second type of data source;
segmenting the synchronization task based on the one or more Reader plugins and Writer plugins to generate a Task queue, the one or more Reader plugins being configured to read data from a source and to send to the Writer plugins and the Writer plugins being configured to write the data to a destination, wherein tasks in the Task queue are processed in parallel; and
implementing the synchronization task using a corresponding number of the Reader plugins and the Writer plugins based on the Task queue.
7. The apparatus of claim 6 , wherein:
the synchronization task is obtained by using a first ClassLoader; and
the synchronization task is implemented by using a second ClassLoader.
8. The apparatus of claim 6 , wherein the one or more Reader plugins are configured to send the data to the Writer plugins via a channel between the Reader plugins and the Writer plugins.
9. The apparatus of claim 6 , wherein the one or more Reader plugins and the Writer plugins each have an independent thread.
10. The apparatus of claim 6 , wherein the one or more Reader plugins and the Writer plugins include init( ), prepare( ), post( ), and destroy( ) methods.
11. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a processor of a computer system, cause the computer system to perform a method, comprising:
obtaining a synchronization task and a corresponding operation classification;
determining one or more Reader plugins and one or more Writer plugins based on the corresponding operation classification, wherein the corresponding operation classification indicates whether the synchronization task is between different types of database systems or between a same type of database system, and wherein the one or more Reader plugins are associated with a first type of data source, the one or more Writer plugins are associated with a second type of data source, and the first type of data source is different from or the same with the second type of data source;
segmenting the synchronization task based on the one or more Reader plugins and Writer plugins to generate a Task queue, the one or more Reader plugins being configured to read data from a source and to send to the Writer plugins and the Writer plugins being configured to write the data to a destination wherein tasks in the Task queue are processed in parallel; and
implementing the synchronization task using a corresponding number of the Reader plugins and the Writer plugins based on the Task queue.
12. The non-transitory computer-readable storage medium of claim 11 , wherein:
the obtaining a synchronization task is performed by a first ClassLoader; and
corresponding number of the Reader plugins and the Writer plugins based on the Task queue are used by a second ClassLoader.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the one or more Reader plugins are configured to send the data to the Writer plugins via a channel between the Reader plugins and the Writer plugins.
14. The non-transitory computer-readable storage medium of claim 11 , wherein the one or more Reader plugins and the Writer plugins each have an independent thread.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the one or more Reader plugins and the Writer plugins include init( ), prepare( ), post( ), and destroy( ), methods.
16. The method of claim 1 , wherein segmenting the synchronization task further comprises:
segmenting the synchronization task based on the one or more Reader plugins;
segmenting the synchronization task based on the one or more of Writer plugins; and
pairing the segmentations to form a plurality of tasks in the Task queue.
17. The method of claim 16 , wherein implementing the synchronization task further comprises:
scheduling the plurality of tasks to different TaskGroups; and
performing synchronization on the TaskGroups based on a parallel start of the plurality of tasks.