Garbage collection of tree structure with page mappings
A key-value engine may perform garbage collection for a tree or hierarchical data structure on an append-only storage device with page mappings. The key-value engine may separate hot and cold data to reduce write amplification, track extent usage using a restricted or limited amount of memory, efficiently answer queries of valid extent usage, and adaptively or selectively defragment pages in snapshots in rounds of garbage collection.
1. A method implemented by one or more processors, the method comprising:
receiving a triggering event to perform garbage collection;
determining one or more extents in a file stream as candidates for the garbage collection based at least in part on an extent usage data structure;
searching the extent usage data structure for a first extent of the one or more extents to obtain a search result, if the first extent is located in a base page stream;
combining the search result with valid ranges identified by snapshot usage page identifiers in the base page stream; and
deleting the one or more extents from the file stream.
2. The method of claim 1 , wherein receiving the triggering event to perform the garbage collection comprises at least one of:
reaching a time set for a garbage collection operation on a regular basis;
detecting that a number of extents ready for the garbage collection is greater than or equal to a predetermined number threshold; or
receiving a user instruction to perform the garbage collection.
3. The method of claim 1 , wherein determining the one or more extents as the candidates for the garbage collection comprises:
determining respective valid data sizes of a plurality of extents using the extent usage data structure; and
selecting a second extent as a candidate for the garbage collection if a total valid data size of the second extent is less than a predetermined size threshold.
4. The method of claim 1 , wherein determining the one or more extents as the candidates for the garbage collection comprises:
determining respective valid data sizes of a plurality of extents using the extent usage data structure; and
ordering the plurality of extents in an ascending order of the respective valid data sizes to form a sorted order of the plurality of extents; and
selecting first N number of extents as the candidates for the garbage collection from the sorted order of the plurality of extents, wherein N is an integer greater than zero.
5. The method of claim 1 , further comprising:
scanning a page mapping data structure to construct valid ranges for a third extent of the one or more extents as of a snapshot timestamp of the garbage collection, if the third extent is located in a delta page stream; and
locating one or more page mapping chains that include valid delta pages.
6. The method of claim 1 , further comprising:
relocating valid pages in the one or more extents; and
updating the extent usage data structure and a page mapping data structure;
recording changes to the extent usage data structure and the page mapping data structure in a log record; and
committing the changes to the extent usage data structure and the page mapping data structure after the changes are recorded in the log record.
7. One or more processor readable media storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform act comprising:
receiving a triggering event to perform garbage collection;
determining one or more extents in a file stream as candidates for the garbage collection based at least in part on an extent usage data structure;
searching the extent usage data structure for a first extent of the one or more extents to obtain a search result, if the first extent is located in a base page stream;
combining the search result with valid ranges identified by snapshot usage page identifiers in the base page stream; and
deleting the one or more extents from the file stream.
8. The one or more processor readable media of claim 7 , wherein receiving the triggering event to perform the garbage collection comprises at least one of:
reaching a time set for a garbage collection operation on a regular basis;
detecting that a number of extents ready for the garbage collection is greater than or equal to a predetermined number threshold; or
receiving a user instruction to perform the garbage collection.
9. The one or more processor readable media of claim 7 , wherein determining the one or more extents as the candidates for the garbage collection comprises:
determining respective valid data sizes of a plurality of extents using the extent usage data structure; and
selecting a second extent as a candidate for the garbage collection if a total valid data size of the second extent is less than a predetermined size threshold.
10. The one or more processor readable media of claim 7 , wherein determining the one or more extents as the candidates for the garbage collection comprises:
determining respective valid data sizes of a plurality of extents using the extent usage data structure; and
ordering the plurality of extents in an ascending order of the respective valid data sizes to form a sorted order of the plurality of extents; and
selecting first N number of extents as the candidates for the garbage collection from the sorted order of the plurality of extents, wherein N is an integer greater than zero.
11. The one or more processor readable media of claim 7 , the acts further comprising:
scanning a page mapping data structure to construct valid ranges for a third extent of the one or more extents as of a snapshot timestamp of the garbage collection, if the third extent is located in a delta page stream; and
locating one or more page mapping chains that include valid delta pages.
12. The one or more processor readable media of claim 11 , the acts further comprising:
relocating valid pages in the one or more extents; and
updating the extent usage data structure and the page mapping data structure;
recording changes to the extent usage data structure and the page mapping data structure in a log record; and
committing the changes to the extent usage data structure and the page mapping data structure after the changes are recorded in the log record.
13. A system comprising:
one or more processors; and
memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving a triggering event to perform garbage collection;
determining one or more extents in a file stream as candidates for the garbage collection based at least in part on an extent usage data structure;
searching the extent usage data structure for a first extent of the one or more extents to obtain a search result, if the first extent is located in a base page stream;
combining the search result with valid ranges identified by snapshot usage page identifiers in the base page stream; and
deleting the one or more extents from the file stream.
14. The system of claim 13 , wherein receiving the triggering event to perform the garbage collection comprises at least one of:
reaching a time set for a garbage collection operation on a regular basis;
detecting that a number of extents ready for the garbage collection is greater than or equal to a predetermined number threshold; or
receiving a user instruction to perform the garbage collection.
15. The system of claim 13 , wherein determining the one or more extents as the candidates for the garbage collection comprises:
determining respective valid data sizes of a plurality of extents using the extent usage data structure; and
selecting a second extent as a candidate for the garbage collection if a total valid data size of the second extent is less than a predetermined size threshold.
16. The system of claim 13 , wherein determining the one or more extents as the candidates for the garbage collection comprises:
determining respective valid data sizes of a plurality of extents using the extent usage data structure; and
ordering the plurality of extents in an ascending order of the respective valid data sizes to form a sorted order of the plurality of extents; and
selecting first N number of extents as the candidates for the garbage collection from the sorted order of the plurality of extents, wherein N is an integer greater than zero.
17. The system of claim 13 , the acts further comprising:
scanning a page mapping data structure to construct valid ranges for a third extent of the one or more extents as of a snapshot timestamp of the garbage collection, if the third extent is located in a delta page stream; and
locating one or more page mapping chains that include valid delta pages.