Scalable and persistent L2 adaptive replacement cache

1. A method for managing file system memory, the method comprising:

a) implementing a separate replacement policy and a separate index for a persistent second level adaptive replacement cache (L2ARC) logically part of a first level ARC;

b) cluster writing compressed chunks of data on multiple physical devices via aligning the clusters of data chunks on a byte boundary basis on each of the devices, thereby creating compressed and packed chunks;

c) creating a storage pool allocator (SPA) to track the compressed and packed chunks on the multiple devices via an active page and multiple closed pages;

d) re-adding an evicted data from the L2ARC to an active page to be written again to the L2ARC based on a configurable threshold number of hits to data in the L2ARC via an L2ARC hit counter; and

e) providing an L2ARC hash table configured to point directly to a data location to enable data to be fetched from one of the multiple devices directly without any requirement to read a header first.

2. The method for managing file system memory of claim 1 , further comprising de-clustering a data collection independently in chunks away from a device used for writing the data in favor of a device for writing a full chunk of data in a scalable write policy.

3. The method for managing file system memory of claim 1 , wherein the replacement policy is based on a sequential and structured log in fixed sized chunks of up to 1 MB each.

4. The method for managing file system memory of claim 1 , wherein the data chunks include logically contiguous data vectors configured with a header to facilitate reading metadata from the multiple devices and to facilitate reading data from an array.

5. The method for managing file system memory of claim 1 , further comprising providing a feed thread configured to scan through a most recently used (MRU) and a most frequently used (MFU) list in the L2ARC and pick up eligible buffers to add to the active page until it is full and written to a device and a new page is started.

6. The method for managing file system memory of claim 1 , further comprising providing a plurality of uberblocks at a fixed location on the multiple devices separate from a list, the uberblocks comprising a pointer to a head of the list and a restore operation to walk the list from the head until it hits a tail indicated in an Uber block.

7. The method for managing file system memory of claim 1 , further comprising resetting the L2ARC hit counter based on adding the evicted data from a physical device to the active page.

8. The method for managing file system memory of claim 1 , further comprising providing a page header buffer and a bucket register file logically disposed between a SPA_Chunk and the hash table.

9. The method for managing file system memory of claim 1 , wherein each multiple device is logically a collection of chunk pages plus Uber pages configured to track a head and a tail of the chunk pages.

10. The method for managing file system memory of claim 9 , wherein the multiple pages include an Uber page chained to another Uber page in a doubly linked list starting from an uber header portion of an Uber page representing where allocation will happen and terminating at an uber tail portion of an Uber page representing a valid range.

11. A system for managing file system memory, the system comprising:

a) a module configured to implement a separate replacement policy and a separate index for a persistent second level adaptive replacement cache (L2ARC) logically part of a first level ARC;

b) a module configured to cluster compressed chunks of data on multiple physical devices via aligning the clusters of data chunks on a byte boundary basis on each device, thereby creating compressed and packed chunks;

c) a storage pool allocator (SPA) configured to track the compressed and packed chunks on the multiple devices via an attached active page and attached multiple closed pages;

d) a module configured to re-add an evicted data from the L2ARC to an active page to be written again to the L2ARC based on a configurable threshold number of hits to data in the L2ARC via an L2ARC hit counter; and

e) an L2ARC hash table configured to point directly to a data location to enable data to be fetched from one of the multiple devices directly without any requirement to read a header first.

12. The system for managing file system memory of claim 11 , further comprising a scalable write module configured to de-cluster data collection independently in chunks away from a device used for writing the data in favor of a most appropriate device for writing a full chunk of data.

13. The system for managing file system memory of claim 11 , wherein the replacement policy is based on a sequential and structured log in fixed sized chunks of up to 1 MB each.

14. The system for managing file system memory of claim 11 , wherein the chunks are logically contiguous data vectors configured to create a header to facilitate reading metadata from the devices and to facilitate reading data from an array.

15. The system for managing file system memory of claim 11 , further comprising a module configured to move an active page to a closed page list based on the active page filling up with data.

16. The system for managing file system memory of claim 11 , wherein each device is logically a collection of chunk pages plus Uber pages configured to track a head and a tail of the chunk pages.

17. The system for managing file system memory of claim 16 , wherein each Uber page is chained to another Uber page in a doubly linked list starting from an uber header portion of an Uber page representing where allocation will happen and terminating at an uber tail portion of an Uber page representing a valid range.

18. A non-transitory processor-readable storage medium having one or more instructions which when executed by at least one processing circuit causes the at least one processing circuit to:

a) implement a separate replacement policy and a separate index for a persistent second level adaptive replacement cache (L2ARC) logically part of a first level ARC;

b) cluster write compressed chunks of data on multiple physical devices in the L2ARC via aligning the clusters of data chunks on a byte boundary basis on each of the devices, thereby creating compressed and packed chunks;

c) create a storage pool allocator (SPA) to track the compressed and packed chunks on the multiple devices via an attached active page and attached multiple closed pages;

d) re-add an evicted data from the L2ARC to an active page to be written again to the L2ARC based on a configurable threshold number of hits to data in the L2ARC via an L2ARC hit counter; and

e) provide an L2ARC hash table configured to point directly to a data location to enable data to be fetched from one of the multiple devices directly without any requirement to read a header first.