System and method for indirect data classification in a storage system operations
A method for managing data includes obtaining data from a host, wherein the data is associated with an object identifier (ID), initiating a classification mapping update to obtain a classification entry, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and a classification ID, storing the storage metadata and the object entry in an accelerator pool, and storing the plurality of deduplicated data chunks and the at least one parity chunk.
1. A method for managing data, the method comprising:
obtaining, by a data classification engine executing on an accelerator pool, the data from a host, wherein the data is associated with an object identifier (ID) and an object, wherein the accelerator pool comprises a first plurality of data nodes;
initiating a classification attributes update to obtain a classification mapping entry,
wherein the classification mapping entry is associated with a classification ID,
wherein a classification mapping comprises the classification mapping entry,
wherein the classification mapping entry of comprises the object ID, the classification ID, and a plurality of classification attributes, and
wherein the plurality of classification attributes comprises at least one of: a retention period for the data and a regulation associated with the data,
wherein the regulation comprises a set of legal standards applied to a legal entity, and
wherein the legal entity owns the data;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks using a unique fingerprint generated for each of the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk;
generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and the classification ID;
storing the storage metadata and the object entry in the accelerator pool; and
storing, across a plurality of fault domains of a non-accelerator pool, the plurality of deduplicated data chunks and the at least one parity chunk,
wherein the non-accelerator pool comprises a second plurality of data nodes, and
wherein the first plurality of data nodes has a higher performance relative to the second plurality of data nodes.
2. The method of claim 1 , further comprising:
making a determination that the classification mapping needs to be changed; and
in response to the determination:
updating the classification mapping to obtain an updated classification mapping; and
updating the object entry based on the update to the classification mapping.
3. The method of claim 2 , wherein initiating the classification attributes update comprises:
obtaining the plurality of classification attributes; and
storing the plurality of classification attributes in the classification mapping.
4. The method of claim 3 , wherein the classification mapping is stored in the data classification engine.
5. The method of claim 1 , further comprising:
initiating metadata distribution of the storage metadata and the object entry across the plurality of fault domains.
6. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data, the method comprising:
obtaining the data from a host, wherein the data is associated with an object identifier (ID) and an object;
initiating a classification attributes update to obtain a classification mapping entry,
wherein the classification mapping entry is associated with a classification ID,
wherein a classification mapping comprises the classification mapping entry,
wherein the classification mapping entry of comprises the object ID, the classification ID, and a plurality of classification attributes, and
wherein the plurality of classification attributes comprises at least one of: a retention period for the data and a regulation associated with the data,
wherein the regulation comprises a set of legal standards applied to a legal entity, and
wherein the legal entity owns the data;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks using a unique fingerprint generated for each of the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk;
generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and the classification ID;
storing the storage metadata and the object entry in the accelerator pool, wherein the accelerator pool comprises a first plurality of data nodes; and
storing, across a plurality of fault domains of a non-accelerator pool, the plurality of deduplicated data chunks and the at least one parity chunk,
wherein the non-accelerator pool comprises a second plurality of data nodes, and
wherein the first plurality of data nodes has a higher performance relative to the second plurality of data nodes.
7. The non-transitory computer readable medium of claim 6 , the method further comprising:
making a determination that the classification mapping needs to be changed; and
in response to the determination:
updating the classification mapping to obtain an updated classification mapping; and
updating the object entry based on the update to the classification mapping.
8. The non-transitory computer readable medium of claim 7 , wherein initiating the classification attributes update comprises:
obtaining the plurality of classification attributes; and
storing the plurality of classification attributes in the classification mapping.
9. The non-transitory computer readable medium of claim 8 , wherein the classification mapping is stored in a data classification engine of the accelerator pool.
10. The non-transitory computer readable medium of claim 6 , the method further comprising:
initiating metadata distribution of the storage metadata and the object entry across the plurality of fault domains.
11. A data cluster, comprising:
a host; and
an accelerator pool comprising a plurality of data nodes,
wherein a data node of the plurality of data nodes comprises a processor and memory comprising instructions, which when executed by the processor perform a method, the method comprising:
obtaining data from the host, wherein the data is associated with an object identifier (ID) and an object;
initiating a classification attributes update to obtain a classification mapping entry,
wherein the classification mapping entry is associated with a classification ID,
wherein a classification mapping comprises the classification mapping entry,
wherein the classification mapping entry of comprises the object ID, the classification ID, and a plurality of classification attributes, and
wherein the plurality of classification attributes comprises at least one of: a retention period for the data and a regulation associated with the data,
wherein the regulation comprises a set of legal standards applied to a legal entity, and
wherein the legal entity owns the data;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks using a unique fingerprint generated for each of the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk;
generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and the classification ID;
storing the storage metadata and the object entry in the accelerator pool; and
storing, across a plurality of fault domains of a non-accelerator pool, the plurality of deduplicated data chunks and the at least one parity chunk,
wherein the non-accelerator pool comprises a second plurality of data nodes, and
wherein the plurality of data nodes has a higher performance relative to the second plurality of data nodes.
12. The data cluster of claim 11 , the method further comprising:
making a determination that the classification mapping needs to be changed; and
in response to the determination:
updating the classification mapping to obtain an updated classification mapping; and
updating the object entry based on the update to the classification mapping.
13. The data cluster of claim 12 , wherein initiating the classification attributes update comprises:
obtaining the plurality of classification attributes; and
storing the plurality of classification attributes in the classification mapping.
14. The data cluster of claim 13 , wherein the classification mapping is stored in a data classification engine of the accelerator pool.
15. The data cluster of claim 11 , the method further comprising:
initiating metadata distribution of the storage metadata and the object entry across the plurality of fault domains.