Rebuilding space accounting counters in mapping layer of storage appliances
Techniques for rebuilding space accounting counters in mapping layers of storage appliances. The techniques include uniquely associating top levels of a mapping layer of a storage appliance with respective storage objects. The techniques further include determining amounts of logical storage space consumed by the respective storage objects from mappings of LBAs of the respective storage objects to virtual blocks of a virtual layer of the storage appliance. The techniques further include determining amounts of physical storage space consumed by the respective storage objects from logged information pertaining to each leaf pointer of a leaf level of the mapping layer that points to a virtual block in the virtual layer, each virtual block being mapped to a physical block in a physical layer of the storage appliance. The techniques further include using multi-threading to determine amounts of logical storage space consumed by dynamically adjustable ranges of the respective storage objects.
1. A method of storage space accounting in a data storage system, comprising:
providing a mapping layer of a storage appliance, the mapping layer including a plurality of block pointer trees, the plurality of block pointer trees including at least a first block pointer tree, the first block pointer tree having a first top level, a first mid-level, and a first leaf level;
implementing data deduplication in a virtual layer of the storage appliance;
uniquely associating the first top level of the first block pointer tree with a first storage object;
determining a first amount of logical storage space consumed by the first storage object from a first mapping of logical block addresses (LBAs) of the first storage object to a plurality of first virtual blocks in the virtual layer in which the data deduplication is implemented;
logging, at intervals, first information pertaining to a plurality of first leaf pointers in the first leaf level of the first block pointer tree that point to the plurality of first virtual blocks in the virtual layer, the logging of the first information including logging, at each interval for each first virtual block, an indication of whether the first virtual block is currently uniquely owned by the first storage object, each first virtual block being mapped to a respective physical block from among a plurality of physical blocks in a physical layer of the storage appliance;
determining a first amount of physical storage space consumed by the first storage object from the logged first information, the determining of the first amount of physical storage space including determining, from the logged indication for each first virtual block, that the plurality of first virtual blocks are currently uniquely owned by the first storage object; and
in response to determining that plurality of first virtual blocks are uniquely owned by the first storage object, deleting the first storage object from the storage appliance or moving the first storage object to a different storage appliance, the deleting or moving of the first storage object including releasing the respective physical block to which each first virtual block is mapped.
2. The method of claim 1 wherein the determining of the first amount of logical storage space consumed by the first storage object includes determining how much of the first storage object is mapped by the first block pointer tree.
3. The method of claim 2 wherein the first top level of the mapping layer includes a first block pointer array, the first mid-level of the mapping layer includes a second block pointer array, and wherein the determining of how much of the first storage object is mapped by the first block pointer tree includes determining how many block pointers in the first block pointer array point to the second block pointer array in the first mid-level of the mapping layer.
4. The method of claim 3 wherein the first leaf level of the mapping layer includes a leaf pointer array, and wherein the determining of how much of the first storage object is mapped by the first block pointer tree includes, having determined how many block pointers in the first block pointer array in the first top level of the mapping layer point to the second block pointer array in the first mid-level of the mapping layer, determining how many block pointers in the second block pointer array in the first mid-level of the mapping layer point to the first leaf pointer array in the first leaf level of the mapping layer.
5. The method of claim 4 wherein the virtual layer includes one or more of the plurality of first virtual blocks that include valid virtual pointers, and wherein the determining of how much of the first storage object is mapped by the first block pointer tree includes, having determined how many block pointers in the second block pointer array in the first mid-level of the mapping layer point to the first leaf pointer array in the first leaf level of the mapping layer, determining how many of the plurality of first leaf pointers in the first leaf pointer array point to the plurality of first virtual blocks that include valid virtual pointers.
6. The method of claim 5 wherein the determining of how much of the first storage object is mapped by the first block pointer tree includes, for each first leaf pointer that points to a respective one of the plurality of first virtual blocks that include valid virtual pointers, logging, in a log data structure, the logged first information pertaining to the plurality of first leaf pointers, the respective one of the plurality of first virtual blocks that include valid virtual pointers, and one or more identifiers corresponding to the first storage object.
7. The method of claim 2 wherein the determining of how much of the first storage object is mapped by the first block pointer tree includes determining, by multi-thread processing of a plurality of storage ranges of the first storage object, how much of the first storage object is mapped by the first block pointer tree.
8. The method of claim 7 wherein the determining of how much of the first storage object is mapped by the first block pointer tree further includes dynamically adjusting the plurality of storage ranges of the first storage object.
9. The method of claim 4 wherein the virtual layer includes one or more of the plurality of first virtual blocks in one or more of a plurality of address ranges that include valid virtual pointers, and wherein the determining of how much of the first storage object is mapped by the block pointer tree includes, having determined how many block pointers in the second block pointer array in the first mid-level of the mapping layer point to the first leaf pointer array in the first leaf level of the mapping layer, determining, for each address range, how many of the plurality of first leaf pointers in the first leaf pointer array point to the plurality of first virtual blocks in the one or more of the plurality of address ranges that include valid virtual pointers.
10. The method of claim 9 wherein the determining of how much of the first storage object is mapped by the first block pointer tree includes, for each first leaf pointer that points to a respective one of the plurality of first virtual blocks in a respective one of the plurality of address ranges, logging, in a log data structure, the logged first information pertaining to the plurality of first leaf pointers, the respective one of the plurality of first virtual blocks in the respective one of the plurality of address ranges that include valid virtual pointers, and one or more identifiers corresponding to the first storage object.
11. The method of claim 10 further comprising:
determining, based on the logged first information for the respective one of the plurality of address ranges, that the first storage object is consuming one or more of the plurality of first virtual blocks in the virtual layer.
12. The method of claim 11 further comprising:
having determined that the plurality of first virtual blocks are uniquely owned by the first storage object, determining an amount of physical storage space consumed by the first storage object based on a mapping between the plurality of first virtual blocks in the respective one of the plurality of address ranges and one or more of the plurality of physical blocks in the physical layer, taking into account how much of the one or more of the plurality of physical blocks are compressed.
13. The method of claim 1 further comprising:
having determined that the plurality of first virtual blocks are uniquely owned by the first storage object, determining an amount of physical storage space consumed by the first storage object based on a mapping between the plurality of first virtual blocks and one or more of the plurality of physical blocks in the physical layer, taking into account how much of the one or more of the plurality of physical blocks are compressed.
14. A system for storage space accounting in a data storage system, comprising:
a memory; and
processing circuitry configured to execute program instructions out of the memory to:
provide a mapping layer of a storage appliance, the mapping layer including a plurality of block pointer trees, the plurality of block pointer trees including at least a first block pointer tree, the first block pointer tree having a first top level, a first mid-level, and a first leaf level;
implementing data deduplication in a virtual layer of the storage appliance;
uniquely associate the first top level of the first block pointer tree with a first storage object;
determine a first amount of logical storage space consumed by the first storage object from a first mapping of logical block addresses (LBAs) of the first storage object to a plurality of first virtual blocks in the virtual layer in which the data deduplication is implemented;
log, at intervals, first information pertaining to a plurality of first leaf pointers in the first leaf level of the first block pointer tree that point to the plurality of first virtual blocks in the virtual layer including logging, at each interval for each first virtual block, an indication of whether the first virtual block is currently uniquely owned by the first storage object, each first virtual block being mapped to a respective physical block from among a plurality of physical blocks in a physical layer of the storage appliance;
determine a first amount of physical storage space consumed by the first storage object from the logged first information including determining, from the logged indication for each first virtual block, that the plurality of first virtual blocks are currently uniquely owned by the first storage object;
and
in response to determining that plurality of first virtual blocks are uniquely owned by the first storage object, deleting the first storage object from the storage appliance or moving the first storage object to a different storage appliance, the deleting or moving of the first storage object including releasing the respective physical block to which each first virtual block is mapped.
15. The system of claim 14 wherein the processing circuitry is further configured to execute the program instructions out of the memory to determine how much of the first storage object is mapped by the first block pointer tree.
16. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method of storage space accounting in a data storage system, the method comprising:
providing a mapping layer of a storage appliance, the mapping layer including a plurality of block pointer trees, the plurality of block pointer trees including at least a first block pointer tree, the first block pointer tree having a first top level, a first mid-level, and a first leaf level;
implementing data deduplication in a virtual layer of the storage appliance;
uniquely associating the first top level of the first block pointer tree with a first storage object;
determining a first amount of logical storage space consumed by the first storage object from a first mapping of logical block addresses (LBAs) of the first storage object to a plurality of first virtual blocks in the virtual layer in which the data deduplication is implemented;
logging, at intervals, first information pertaining to a plurality of first leaf pointers in the first leaf level of the first block pointer tree that point to the plurality of first virtual blocks in the virtual layer, the logging of the first information including logging, at each interval for each first virtual block, an indication of whether the first virtual block is currently uniquely owned by the first storage object, each first virtual block being mapped to a respective physical block from among a plurality of physical blocks in a physical layer of the storage appliance;
determining a first amount of physical storage space consumed by the first storage object from the logged first information, the determining of the first amount of physical storage space including determining, from the logged indication for each first virtual block, that the plurality of first virtual blocks are currently uniquely owned by the first storage object;
and
in response to determining that plurality of first virtual blocks are uniquely owned by the first storage object, deleting the first storage object from the storage appliance or moving the first storage object to a different storage appliance, the deleting or moving of the first storage object including releasing the respective physical block to which each first virtual block is mapped.
17. The computer program product of claim 16 wherein the determining of the first amount of logical storage space consumed by the first storage object includes determining how much of the first storage object is mapped by the first block pointer tree.