Persistent cache layer to tier data to cloud storage
Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods as a part of tiering data to external repositories. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.
1. A method comprising:
establishing at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establishing a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a set of cloud storage metadata; and
processing file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer, wherein processing file system operations includes:
processing a read operation based on a state of the cache overlay layer, and
processing a write operation by writing the write operation to the cache overlay layer and marking the chunk state for each chunk of file data implicated by the write operation.
2. The method of claim 1 , wherein the set of cloud storage metadata includes at least one of a cloud metadata object and a cloud data object.
3. The method of claim 1 , wherein processing file system operations received from the file system client includes processing a read operation targeted to a file, wherein processing the read operation includes:
determining a set of cloud data objects associated with the file based on the set of cloud storage metadata associated with the storage layer inode of the file;
retrieving a subset of the set of cloud data objects from a cloud storage location based on the set of cloud storage metadata; and
filling the subset of the set of cloud data objects into the cache overlay.
4. The method of claim 3 , further comprising:
in response to retrieving the subset of the set of cloud data objects, at least one of decompressing or decrypting the subset of the set of cloud objects.
5. The method of claim 1 , wherein processing file system operations includes processing a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.
6. The method of claim 1 , wherein processing file system operations includes processing a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
generating a modified set of cloud storage metadata based on the identified set of chunks;
associating the modified set of cloud storage metadata with the storage layer inode;
storing the set of chunks and the modified set of cloud storage metadata in a cloud storage location based on the modified set of cloud storage metadata; and
changing the chunk state of the set of chunks to clean.
7. The method of claim 6 , further comprising:
in response to identifying the set of chunks, at least one of encrypting or compressing the set of chunks based on the set of cloud storage metadata, wherein generating the modified set of cloud storage metadata is further based on the at least one encrypting or compressing.
8. A system comprising at least one storage device and at least one hardware processor configured to:
establish at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establish a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a set of cloud storage metadata, wherein the cloud storage metadata includes at least one or more of a reference to a cloud metadata; and
process file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer, wherein processing file system operations includes:
processing a read operation based on a state of the cache overlay layer, and
processing a write operation by writing the write operation to the cache overlay layer and marking the chunk state for each chunk of file data implicated by the write operation.
9. The system of claim 8 , wherein the set of cloud storage metadata includes at least one of a cloud metadata object and a cloud data object.
10. The system of claim 8 , further configured to:
process a read operation from the file system client targeted to a file, wherein processing the read operation includes:
determining a set of cloud data objects associated with the file based on the set of cloud storage metadata associated with the storage layer inode of the file;
retrieving a subset of the set of cloud data objects from a cloud storage location based on the set of cloud storage metadata; and
filling the subset of the set of cloud data objects into the cache overlay.
11. The system of claim 10 , further configured to:
in response to retrieving the subset of the set of cloud data objects, at least one of decompressing or decrypting the subset of the set of cloud objects.
12. The system of claim 8 , further configured to:
process a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.
13. The system of claim 8 , further configured to:
Process a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
generating a modified set of cloud storage metadata based on the identified set of chunks;
associating the modified set of cloud storage metadata with the storage layer inode;
storing the set of chunks and the modified set of cloud storage metadata in a cloud storage location based on the modified set of cloud storage metadata; and
changing the chunk state of the set of chunks to clean.
14. The system of claim 13 , further configured to:
in response to identifying the set of chunks, at least one of encrypt or compress the set of chunks based on the set of cloud storage metadata, wherein generating the modified set of cloud storage metadata is further based on the at least one encrypting or compressing.
15. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
establishing at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establishing a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a set of cloud storage metadata; and
processing file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer, wherein processing file system operations includes:
processing a read operation based on a state of the cache overlay layer, and
processing a write operation by writing the write operation to the cache overlay layer and marking the chunk state for each chunk of file data implicated by the write operation.
16. The non-transitory computer readable medium of claim 15 , wherein the set of cloud storage metadata includes at least one of a cloud metadata object and a cloud data object.
17. The non-transitory computer readable medium of claim 15 , wherein processing file system operations received from the file system client includes processing a read operation targeted to a file, wherein processing the read operation includes:
determining a set of cloud data objects associated with the file based on the set of cloud storage metadata associated with the storage layer inode of the file;
retrieving a subset of the set of cloud data objects from a cloud storage location based on the set of cloud storage metadata; and
filling the subset of the set of cloud data objects into the cache overlay.
18. The non-transitory computer readable medium of claim 15 , with program instructions stored thereon to further perform the following acts:
in response to retrieving the subset of the set of cloud data objects, at least one of decompressing or decrypting the subset of the set of cloud objects.
19. The non-transitory computer readable medium of claim 15 , wherein processing file system operations includes processing a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.
20. The non-transitory computer readable medium of claim 15 , wherein processing file system operations includes processing a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
generating a modified set of cloud storage metadata based on the identified set of chunks;
associating the modified set of cloud storage metadata with the storage layer inode;
storing the set of chunks and the modified set of cloud storage metadata in a cloud storage location based on the modified set of cloud storage metadata; and
changing the chunk state of the set of chunks to clean.