Method and system for performing data deduplication in a data pipeline
In general, the invention relates to a method for processing data. The method includes receiving a write request from a host, and in response to the write request, obtaining system metadata for a system, selecting, based on the system metadata, a selected component of the system to perform a data processing operation, and initiating the data processing operation on the selected component.
1. A method for processing data in a system, the method comprising:
receiving, by a data manager, a write request from a host, wherein the write request specifies the data, wherein components of the system comprise the host, the data manager, a data cluster, and a remote cluster;
in response to the write request, obtaining, by the data manager, system metadata associated with each of the components, wherein obtaining the system metadata comprises:
sending a request to the host to obtain host metadata,
sending a request to the data cluster to obtain data cluster metadata, and
sending a request to the remote cluster to obtain remote cluster metadata,
wherein the system metadata comprises the host metadata, the data cluster metadata, and the remote cluster metadata;
selecting, based on the system metadata, a selected component of the components to perform a data processing operation, wherein the system metadata specifies that the selected component is capable of performing the data processing operation;
initiating transfer of the data from the host to the selected component;
initiating the data processing operation on the selected component, wherein the selected components applies the data processing operation to the data to generate processed data and wherein the processed data is stored in one selected from a group consisting of the data cluster and the remote cluster.
2. The method of claim 1 , wherein the selected component is a network element.
3. The method of claim 2 , wherein the system metadata further comprises network metadata obtained from the network element.
4. The method of claim 1 , wherein the data processing operation is a deduplication operation.
5. A system, comprising:
a processor;
a data manager, which when executed by the processor performs a method, the method comprising:
receiving a write request from a host, wherein the write request specifies data, wherein components of the system further comprise the host a data cluster, and a remote cluster;
in response to the write request, obtaining, system metadata associated with each of the components, wherein obtaining the system metadata comprises:
sending a request to the host to obtain host metadata,
sending a request to the data cluster to obtain data cluster metadata, and
sending a request to the remote cluster to obtain remote cluster metadata,
wherein the system metadata comprises the host metadata, the data cluster metadata, and the remote cluster metadata;
selecting, based on the system metadata, a selected component of the components to perform a data processing operation, wherein the system metadata specifies that the selected component is capable of performing the data processing operation;
initiating transfer of the data from the host to the selected component;
initiating the data processing operation on the selected component, wherein the selected components applies the data processing operation to the data to generate processed data and wherein the processed data is stored in one selected from a group consisting of the data cluster and the remote cluster.
6. The system of claim 5 , wherein the selected component is a network element.
7. The system of claim 6 , wherein the system metadata further comprises network metadata obtained from the network element.
8. The system of claim 5 , wherein the data processing operation is a deduplication operation.
9. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:
receiving, by a data manager, a write request from a host, wherein the write request specifies the data, wherein components of a system comprise the host, the data manager, a data cluster, and a remote cluster;
in response to the write request, obtaining, by the data manager, system metadata associated with each of the components, wherein obtaining the system metadata comprises:
sending a request to the host to obtain host metadata,
sending a request to the data cluster to obtain data cluster metadata, and
sending a request to the remote cluster to obtain remote cluster metadata,
wherein the system metadata comprises the host metadata, the data cluster metadata, and the remote cluster metadata;
selecting, based on the system metadata, a selected component of the components to perform a data processing operation, wherein the system metadata specifies that the selected component is capable of performing the data processing operation;
initiating transfer of the data from the host to the selected component;
initiating the data processing operation on the selected component, wherein the selected components applies the data processing operation to the data to generate processed data and wherein the processed data is stored in one selected from a group consisting of the data cluster and the remote cluster.
10. The non-transitory computer readable medium of claim 9 , wherein the selected component is a network element.
11. The non-transitory computer readable medium of claim 10 , wherein the system metadata further comprises network metadata obtained from the network element.