End-of-life data page processing
A method of managing end-of-life processing of data blocks includes performing a decref procedure for a chunk of data, the decref procedure decrementing reference counts for respective blocks of the chunk of data, and deferring performance of an end-of-life procedure for a plurality of blocks of the chunk of data whose reference counts become zero as a result of performing the decref procedure. The method further includes, upon performing a data consolidation procedure after performing the decref procedure, performing the end-of-life procedure on the plurality of blocks.
1 . A method of managing end-of-life processing of data blocks, comprising:
performing a decref procedure for a chunk of data, the decref procedure decrementing reference counts for respective blocks of the chunk of data;
deferring performance of an end-of-life procedure for a plurality of blocks of the chunk of data whose reference counts become zero as a result of performing the decref procedure; and
upon performing a data consolidation procedure on the chunk of data after performing the decref procedure, performing the end-of-life procedure on the plurality of blocks,
wherein performing the end-of-life procedure includes accessing the chunk of data and an associated chunk of metadata that are already in memory for performance of the data consolidation procedure, without having to load the chunk of data or the chunk of metadata separately from persistent storage.
2 . The method of claim 1 ,
wherein performing the data consolidation procedure includes loading both the chunk of data and the associated chunk of metadata into memory from persistent storage, the chunk of metadata storing the reference counts of the blocks of the chunk of data, the chunk of data and the chunk of metadata being in memory enabling the data consolidation procedure to consolidate the chunk of data into a new chunk of data.
3 . The method of claim 2 , wherein performing the data consolidation procedure includes combining still-valid blocks from the chunk of data with still-valid blocks from at least one other chunk of data and placing the combined blocks into the new chunk of data.
4 . The method of claim 2 , wherein performing the data consolidation procedure includes combining still-valid blocks from the chunk of data with new blocks of data and placing the combined blocks into the new chunk of data.
5 . The method of claim 2 , wherein performing the end-of-life procedure includes:
identifying the plurality of blocks based on the reference counts in the chunk of metadata;
computing digests of the blocks of the plurality of blocks;
performing lookups for entries in a deduplication database based on the computed digests; and
for one or more entries located by performing the lookups, removing said one or more entries from the deduplication database.
6 . The method of claim 5 , wherein identifying the plurality of blocks includes scanning the reference counts in the chunk of metadata and constructing a bitmap that associates blocks having reference counts of zero with associated digests.
7 . The method of claim 5 , wherein the deduplication database is maintained at least in part in memory, such that removing said one or more entries from the deduplication database frees portions of memory such that those portions are able to store new entries.
8 . The method of claim 5 , wherein the plurality of blocks includes multiple compressed blocks, and wherein performing the end-of-life procedure further includes decompressing the compressed blocks before computing the digests of those blocks.
9 . The method of claim 8 ,
wherein performing the data consolidation procedure includes decompressing multiple extents of the chunk of data, each of the extents including multiple blocks, and, for at least one of the extents, identifying invalid blocks within the extent, recompressing data of the extent without the invalid blocks, and writing the recompressed data of the extent to the new chunk of data, and
wherein computing the digest of each block is based on a decompressed extent that was decompressed as part of the data consolidation procedure for at least a subset of the plurality of blocks.
10 . A computerized apparatus, comprising control circuitry that includes a set of processors coupled to memory, the control circuitry constructed and arranged to:
perform a decref procedure for a chunk of data, the decref procedure decrementing reference counts for respective blocks of the chunk of data;
defer performance of an end-of-life procedure for a plurality of blocks of the chunk of data whose reference counts become zero as a result of performing the decref procedure; and
upon performing a data consolidation procedure on the chunk of data after performing the decref procedure, perform the end-of-life procedure on the plurality of blocks,
wherein the control circuitry constructed and arranged to perform the end-of-life procedure is further constructed and arranged to access the chunk of data and an associated chunk of metadata that are already in memory for performance of the data consolidation procedure, without having to load the chunk of data or the chunk of metadata separately from persistent storage.
11 . The computerized apparatus of claim 10 ,
wherein the control circuitry constructed and arranged to perform the data consolidation procedure is further constructed and arranged to load both the chunk of data and the associated chunk of metadata into memory from persistent storage, the chunk of metadata storing the reference counts of the blocks of the chunk of data, the chunk of data and the chunk of metadata being in memory enabling the data consolidation procedure to consolidate the chunk of data into a new chunk of data.
12 . A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of managing end-of-life processing of data blocks, the method comprising:
performing a decref procedure for a chunk of data, the decref procedure decrementing reference counts for respective blocks of the chunk of data;
deferring performance of an end-of-life procedure for a plurality of blocks of the chunk of data whose reference counts become zero as a result of performing the decref procedure; and
upon performing a data consolidation procedure on the chunk of data after performing the decref procedure, performing the end-of-life procedure on the plurality of blocks,
wherein performing the end-of-life procedure includes accessing the chunk of data and an associated chunk of metadata that are already in memory for performance of the data consolidation procedure, without having to load the chunk of data or the chunk of metadata separately from persistent storage.
13 . The computer program product of claim 12 ,
wherein performing the data consolidation procedure includes loading both the chunk of data and the associated chunk of metadata into memory from persistent storage, the chunk of metadata storing the reference counts of the blocks of the chunk of data, the chunk of data and the chunk of metadata being in memory enabling the data consolidation procedure to consolidate the chunk of data into a new chunk of data.
14 . The computer program product of claim 13 , wherein performing the data consolidation procedure includes combining still-valid blocks from the chunk of data with still-valid blocks from at least one other chunk of data and placing the combined blocks into the new chunk of data.
15 . The computer program product of claim 13 , wherein performing the data consolidation procedure includes combining still-valid blocks from the chunk of data with new blocks of data and placing the combined blocks into the new chunk of data.
16 . The computer program product of claim 13 , wherein performing the end-of-life procedure includes:
identifying the plurality of blocks based on the reference counts in the chunk of metadata;
computing digests of the blocks of the plurality of blocks;
performing lookups for entries in a deduplication database based on the computed digests; and
for one or more entries located by performing the lookups, removing said one or more entries from the deduplication database.
17 . The computer program product of claim 16 , wherein identifying the plurality of blocks includes scanning the reference counts in the chunk of metadata and constructing a bitmap that associates blocks having reference counts of zero with associated digests.
18 . The computer program product of claim 16 , wherein the deduplication database is maintained at least in part in memory, such that removing said one or more entries from the deduplication database frees portions of memory such that those portions are able to store new entries.
19 . The computer program product of claim 16 , wherein the plurality of blocks includes multiple compressed blocks, and wherein performing the end-of-life procedure further includes decompressing the compressed blocks before computing the digests of those blocks.
20 . The computer program product of claim 19 ,
wherein performing the data consolidation procedure includes decompressing multiple extents of the chunk of data, each of the extents including multiple blocks, and, for at least one of the extents, identifying invalid blocks within the extent, recompressing data of the extent without the invalid blocks, and writing the recompressed data of the extent to the new chunk of data, and
wherein computing the digest of each block is based on a decompressed extent that was decompressed as part of the data consolidation procedure for at least a subset of the plurality of blocks.