Data de-duplication for information storage systems
View Patent ↗Technologies for eliminating duplicate data within a storage system. De-duplication may be performed done at physical chunk level, where the data is not copied or moved to different location. A logical mapping is modified using a thin de-duplication kernel module that resides between a distributed volume manager (DVM) and a logical disk (LD). De-duplication is achieved by changing pointers in the mapping to land at a physical location. De-duplication is performed as post-process feature where duplicates are indentified and the duplicates are marked in the mapping table, thereby claiming free space through de-duplication. Block-level de-duplication in accordance with the above can co-exist with existing storage architectures for thin provisioning and snapshot management.
1. A method for removing duplicate data within a data storage system, comprising:
identifying duplicated data within the data storage system;
initializing a de-duplication table;
creating at least one circular list of indices in the de-duplication table, wherein each of the indices points to a physical location having a same data;
receiving I/O requests; and
mapping the I/O requests to the physical location using the de-duplication table.
2. The method of claim 1 , further comprising avoiding moving any data movement as part of the removing duplicate.
3. The method of claim 1 , further comprising initializing the de-duplication table at a point in time when the duplicated data is identified.
4. The method of claim 1 , further comprising performing the data de-duplication at a chunk level.
5. The method of claim 1 , further comprising:
walking the at least one circular list to a next entry associated with an index to which a read I/O request is received; and
retrieving data from the physical location associated with the next entry.
6. The method of claim 5 , further comprising:
accessing a status bit associated with the index to which the read I/O request is received; and
accessing a phantom LD table if a foreign status bit is set in order to obtain the data.
7. The method of claim 1 , further comprising:
logging updates to the de-duplication table in a log file; and
persisting the log file to disk.
8. The method of claim 7 , further comprising replaying the log file to recreate the de-duplication table in the event of a power-cycle.
9. The method of claim 1 , further comprising performing the method as a post-processing operation after I/Os are received.
10. The method of claim 1 , further comprising:
providing the de-duplication table as a group of tables, wherein each group is a size of a memory page; and
allocating a page in memory a zone within a group is identified as having duplicated data.
11. A data storage system, comprising:
a storage server;
a physical storage device associated with the storage server;
a processing unit associated with the storage server; and
one or more modules for execution on the processing unit, operable to identify duplicated data within the data storage system, initialize a de-duplication table, create at least one circular list of indices in the de-duplication table, wherein each of the indices points to a physical location having a same data, receiving I/O requests, and map the I/O requests to the physical location using the de-duplication table.
12. The data storage system of claim 11 , further comprising avoiding moving any data movement as part of the removing duplicate.
13. The data storage system of claim 11 , further comprising initializing the de-duplication table at a point in time when the duplicated data is identified.
14. The data storage system of claim 11 , further comprising:
walking the at least one circular list to a next entry associated with an index to which a read I/O request is received; and
retrieving data from the physical location associated with the next entry.
15. The data storage system of claim 14 , further comprising:
accessing a status bit associated with the index to which the read I/O request is received; and
accessing a phantom LD table if a foreign status bit is set in order to obtain the data.
16. The data storage system of claim 11 , further comprising:
logging updates to the de-duplication table in a log file; and
persisting the log file to disk.
17. The data storage system of claim 11 , further comprising performing the method as a post-processing operation after I/Os are received.
18. The data storage system of claim 11 , further comprising:
providing the de-duplication table as a group of tables, wherein each group is a size of a memory page; and
allocating a page in memory a zone within a group is identified as having duplicated data.
19. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a computer system, cause the computer system to:
identify duplicated data within the data storage system;
initialize a de-duplication table;
create at least one circular list of indices in the de-duplication table, wherein each of the indices points to a physical location having a same data;
receiving I/O requests; and
map the I/O requests to the physical location using the de-duplication table.
20. The non-transitory computer-readable storage medium of claim 19 , further comprising causing the computer system to walk the at least one circular list to a next entry associated with an index to which a read I/O request is received; and retrieve data from the physical location associated with the next entry.