METHOD AND SYSTEM FOR PERFORMING DATA DEDUPLICATION AND COMPRESSION IN A DATA CLUSTER
In general, the invention relates to a method for storing data. The method includes obtaining data from a host, making a first determination that a data cluster comprises a plurality of non-volatile memory devices, based on the first determination: storing processed data corresponding to the data in at least one of the plurality of non-volatile memory devices, and making a second determination to de-stage at least a portion of the processed data to a persistent storage device, and based on the second determination, initiating a delayed compression operation on the processed data.
1 . A method for storing data, the method comprising:
obtaining host data from a host;
making a first determination that a data cluster comprises a plurality of non-volatile memory devices;
based on the first determination:
storing processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices, wherein the processed data is generated based on a protection policy specifying requirements for generating and storing the processed data associated with the host data; and
making a second determination to de-stage at least a portion of the processed data to a persistent storage device; and
based on the second determination, initiating a delayed compression operation on the processed data.
2 . The method of claim 1 , wherein the delayed compression operation comprises:
compressing at least a portion of the processed data to obtain compressed data; and
initiating transmission of the compressed data to the persistent storage device.
3 . The method of claim 1 , wherein storing the processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices comprising:
performing an erasure coding operation on at least a portion of the host data to generate processed data; and
initiating transmission of the processed data to the at least one of the plurality of non-volatile memory devices.
4 . The method of claim 3 , wherein performing the erasure coding operation comprises generating at least one parity value using at least a portion of the host data.
5 . The method of claim 1 , wherein storing the processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices comprising:
performing a replication operation on at least a portion of the host data to generated processed data; and
initiating transmission of the processed data to the at least one of the plurality of non-volatile memory devices.
6 . The method of claim 1 , wherein the data cluster comprises at least one node, wherein the at least one of the plurality of non-volatile memory devices and the persistent storage device are located on the at least one node.
7 . The method of claim 1 , wherein making the second determination to de-stage at least a portion of the processed data to the persistent storage device comprises:
making a third determination that a storage threshold for the at least one of the plurality of non-volatile memory devices has been exceeded.
8 . A system, comprising:
a processor;
a data processor, which when executed by the processor performs a method, the method comprising:
obtaining host data from a host;
making a first determination that a data cluster comprises a plurality of non-volatile memory devices;
based on the first determination:
storing processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices, wherein the processed data is generated based on a protection policy specifying requirements for generating and storing the processed data associated with the host data; and
making a second determination to de-stage at least a portion of the processed data to a persistent storage device; and
based on the second determination, initiating a delayed compression operation on the processed data.
9 . The system of claim 8 , wherein the delayed compression operation comprises:
compressing at least a portion of the processed data to obtain compressed data; and
initiating transmission of the compressed data to the persistent storage device.
10 . The system of claim 8 , wherein storing the processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices comprising:
performing an erasure coding operation on at least a portion of the host data to generate processed data; and
initiating transmission of the processed data to the at least one of the plurality of non-volatile memory devices.
11 . The system of claim 10 , wherein performing the erasure coding operation comprises generating at least one parity value using at least a portion of the host data.
12 . The system of claim 8 , wherein storing the processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices comprising:
performing a replication operation on at least a portion of the host data to generated processed data; and
initiating transmission of the processed data to the at least one of the plurality of non-volatile memory devices.
13 . The system of claim 8 , wherein the data cluster comprises at least one node, wherein the at least one of the plurality of non-volatile memory devices and the persistent storage device are located on the at least one node.
14 . The system of claim 8 , wherein making the second determination to de-stage at least a portion of the processed data to the persistent storage device comprises:
making a third determination that a storage threshold for the at least one of the plurality of non-volatile memory devices has been exceeded.
15 . A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:
obtaining host data from a host;
making a first determination that a data cluster comprises a plurality of non-volatile memory devices;
based on the first determination:
storing processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices, wherein the processed data is generated based on a protection policy specifying requirements for generating and storing the processed data associated with the host data; and
making a second determination to de-stage at least a portion of the processed data to a persistent storage device; and
based on the second determination, initiating a delayed compression operation on the processed data.
16 . The non-transitory computer readable medium of claim 15 , wherein the delayed compression operation comprises:
compressing at least a portion of the processed data to obtain compressed data; and
initiating transmission of the compressed data to the persistent storage device.
17 . The non-transitory computer readable medium of claim 15 , wherein storing the processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices comprising:
performing an erasure coding operation on at least a portion of the host data to generate processed data; and
initiating transmission of the processed data to the at least one of the plurality of non-volatile memory devices.
18 . The non-transitory computer readable medium of claim 17 , wherein performing the erasure coding operation comprises generating at least one parity value using at least a portion of the host data.
19 . The non-transitory computer readable medium of claim 15 , wherein storing the processed data corresponding to the host data in at least one of the plurality of non-volatile memory devices comprising:
performing a replication operation on at least a portion of the host data to generated processed data; and
initiating transmission of the processed data to the at least one of the plurality of non-volatile memory devices.
20 . The non-transitory computer readable medium of claim 15 , wherein the data cluster comprises at least one node, wherein the at least one of the plurality of non-volatile memory devices and the persistent storage device are located on the at least one node.