Managing inline data de-duplication in storage systems
A method is used in managing inline data de-duplication in storage systems. The method receives a request to write data at a logical address of a file in a file system of a storage system. The method determines whether the data can be de-duplicated to matching data residing on the storage system in a compressed format. Based on the determination, the method uses a block mapping pointer associated with the matching data to de-duplicate the data. The block mapping pointer includes a block mapping of a set of compressed data extents and information regarding location of the matching data within the set of compressed data extents.
1. A method for use in managing inline data de-duplication in storage systems, the method comprising:
receiving a request to write data at a logical address of a file in a file system of a storage system;
determining whether the data can be de-duplicated to matching data residing on the storage system in a compressed format; and
based on the determination, using a block mapping pointer associated with the matching data to de-duplicate the data, wherein the block mapping pointer includes a block mapping of a set of compressed data extents and information regarding location of the matching data within the set of compressed data extents, wherein, to add de-duplication to a pre-existing inline compression process, the block mapping pointer associated with the matching data to be de-duplicated is a de-duplication mapping pointer comprised of an inline compression mapping pointer and extent index information indicating a location of matching compressed data and a block mapping pointer structure used for inline compression includes a block mapping type indicating de-duplication.
2. The method of claim 1 , further comprising:
indicating the data is de-duplicated to the matching data within the set of compressed data extents by pointing to the block mapping by a de-duplication block mapping pointer; and
identifying, in an indirect block, the de-duplication block mapping pointer, wherein the de-duplication block mapping pointer is stored in the indirect block at an offset.
3. The method of claim 2 , wherein the de-duplication block mapping pointer comprises a block mapping type indicating de-duplication.
4. The method of claim 3 , further comprising:
in response to a request to read the matching data pointed to by the de-duplication block mapping pointer located at the offset of the indirect block, upon identifying that the block mapping type indicates de-duplication, reading an extent index in the de-duplication block mapping pointer to determine the location of the matching data within the set of compressed data extents.
5. The method of claim 1 , further comprising:
in response to de-duplicating the data, incrementing a distribution weight associated with the matching data within the set of compressed data extents in the block mapping, wherein the distribution weight is maintained in the block mapping.
6. The method of claim 1 , wherein the block mapping pointer is a virtual block mapping pointer.
7. The method of claim 1 , wherein determining whether the data can be de-duplicated to the matching data residing on the storage system in the compressed format comprises:
performing a lookup, in a digest cache, to determine if a matching digest key associated with the matching data matches a digest key associated with the data; and
upon identifying the matching digest key, obtaining the block mapping associated with the matching digest key, wherein the digest cache comprises the block mapping and an extent index indicating the location of the matching data within the set of compressed data extents.
8. The method of claim 7 , further comprising:
creating the de-duplication block mapping pointer comprising the block mapping and the extent index.
9. A system for use in managing inline data de-duplication in storage systems, the system comprising a processor configured to:
receive a request to write data at a logical address of a file in a file system of a storage system;
determine whether the data can be de-duplicated to matching data residing on the storage system in a compressed format; and
based on the determination, use a block mapping pointer associated with the matching data to de-duplicate the data, wherein the block mapping pointer includes a block mapping of a set of compressed data extents and information regarding location of the matching data within the set of compressed data extents, wherein, to add de-duplication to a pre-existing inline compression process, the block mapping pointer associated with the matching data to be de-duplicated is a de-duplication mapping pointer comprised of an inline compression mapping pointer and extent index information indicating a location of matching compressed data and a block mapping pointer structure used for inline compression includes a block mapping type indicating de-duplication.
10. The system of claim 9 , further configured to:
indicate the data is de-duplicated to the matching data within the set of compressed data extents by pointing to the block mapping by a de-duplication block mapping pointer; and
identify, in an indirect block, the de-duplication block mapping pointer, wherein the de-duplication block mapping pointer is stored in the indirect block at an offset.
11. The system of claim 10 , wherein the de-duplication block mapping pointer comprises a block mapping type indicating de-duplication.
12. The system of claim 11 , further configured to:
in response to a request to read the matching data pointed to by the de-duplication block mapping pointer located at the offset of the indirect block, upon identifying that the block mapping type indicates de-duplication, read an extent index in the de-duplication block mapping pointer to determine the location of the matching data within the set of compressed data extents.
13. The system of claim 9 , further configured to:
in response to de-duplicating the data, increment a distribution weight associated with the matching data within the set of compressed data extents in the block mapping, wherein the distribution weight is maintained in the block mapping.
14. The system of claim 9 , wherein the block mapping pointer is a virtual block mapping pointer.
15. The system of claim 9 , wherein the processor configured to determine whether the data can be de-duplicated to the matching data residing on the storage system in the compressed format is further configured to:
perform a lookup, in a digest cache, to determine if a matching digest key associated with the matching data matches a digest key associated with the data; and
upon identifying the matching digest key, obtain the block mapping associated with the matching digest key, wherein the digest cache comprises the block mapping and an extent index indicating the location of the matching data within the set of compressed data extents.
16. The system of claim 15 , further configured to:
create the de-duplication block mapping pointer comprising the block mapping and the extent index.
17. A computer program product for inline data de-duplication in storage systems, the computer program product comprising:
a computer readable storage medium having computer executable program code embodied therewith, the program code executable by a computer processor to:
receive a request to write data at a logical address of a file in a file system of a storage system;
determine whether the data can be de-duplicated to matching data residing on the storage system in a compressed format; and
based on the determination, use a block mapping pointer associated with the matching data to de-duplicate the data, wherein the block mapping pointer includes a block mapping of a set of compressed data extents and information regarding location of the matching data within the set of compressed data extents, wherein, to add de-duplication to a pre-existing inline compression process, the block mapping pointer associated with the matching data to be de-duplicated is a de-duplication mapping pointer comprised of an inline compression mapping pointer and extent index information indicating a location of matching compressed data and a block mapping pointer structure used for inline compression includes a block mapping type indicating de-duplication.
18. The computer program product of claim 17 , the program code further configured to:
indicate the data is de-duplicated to the matching data within the set of compressed data extents by pointing to the block mapping by a de-duplication block mapping pointer; and
identify, in an indirect block, the de-duplication block mapping pointer, wherein the de-duplication block mapping pointer is stored in the indirect block at an offset.
19. The computer program product of claim 18 , wherein the de-duplication block mapping pointer comprises a block mapping type indicating de-duplication.
20. The computer program product of claim 18 , wherein the program code configured to determine whether the data can be de-duplicated to the matching data residing on the storage system in the compressed format is further configured to:
perform a lookup, in a digest cache, to determine if a matching digest key associated with the matching data matches a digest key associated with the data; and
upon identifying the matching digest key, obtain the block mapping associated with the matching digest key, wherein the digest cache comprises the block mapping and an extent index indicating the location of the matching data within the set of compressed data extents.