Persistent cache layer in a distributed file system
Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods or store unmodified raw data in local storage. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.
1. A method comprising:
establishing at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establishing a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a data augmentation group; and
processing file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer.
2. The method of claim 1 , wherein the data augmentation group is compression, and wherein the second data stream is compressed in the storage layer.
3. The method of claim 1 , wherein the data augmentation group is encryption, and wherein the second data stream is encrypted in the storage layer.
4. The method of claim 1 , wherein processing file system operations received from the file system client includes processing a write operation, wherein processing the write operation includes:
writing file data associated with the write operation to the cache overlay; and
marking the chunk state in the cache overlay inode as dirty for each chunk of file data implicated by the write operation.
5. The method of claim 1 , wherein processing file system operations received from the file system client includes processing a read operation targeted to a file, wherein processing the read operation includes:
determining the data augmentation group associated with the storage layer inode of the file;
transforming a set of file data of the file stored in the storage layer based on the data augmentation group and a set of storage layer metadata referenced by the storage layer inode of the file; and
filling the transformed set of file data into the cache overlay from the storage layer.
6. The method of claim 1 , wherein processing file system operations includes processing a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.
7. The method of claim 1 , wherein processing file system operations includes processing a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
augmenting the set of chunks based on the data augmentation group associated with the storage layer inode of the file;
generating a set of augmentation metadata based on the augmenting;
storing the augmented set of chunks and the set of augmentation metadata in the storage layer based on the storage layer inode of the file; and
changing the chunk state of the set of chunks to clean.
8. A system comprising at least one storage device and at least one hardware processor configured to:
establish at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establish a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a data augmentation group; and
process file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer.
9. The system of claim 8 , wherein the data augmentation group is compression, and wherein the second data stream is compressed in the storage layer.
10. The system of claim 8 , wherein the data augmentation group is encryption, and wherein the second data stream is encrypted in the storage layer.
11. The system of claim 8 , further configured to:
process a write operation received from the file system client, and wherein processing the write operation includes:
writing file data associated with the write operation to the cache overlay; and
marking the chunk state in the cache overlay inode as dirty for each chunk of file data implicated by the write operation.
12. The system of claim 8 , further configured to:
process a read operation received from the file system client targeted to a file, wherein processing the read operation includes:
determining the data augmentation group associated with the storage layer inode of the file;
transforming a set of file data of the file stored in the storage layer based on the data augmentation group and a set of storage layer metadata referenced by the storage layer inode of the file; and
filling the transformed set of file data into the cache overlay from the storage layer.
13. The system of claim 8 , further configured to:
process a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.
14. The system of claim 8 , further configured to:
process a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
augmenting the set of chunks based on the data augmentation group associated with the storage layer inode of the file;
generating a set of augmentation metadata based on the augmenting;
storing the augmented set of chunks and the set of augmentation metadata in the storage layer based on the storage layer inode of the file; and
changing the chunk state of the set of chunks to clean.
15. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
establishing at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establishing a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a data augmentation group; and
processing file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer.
16. The non-transitory computer readable medium of claim 15 , wherein the data augmentation group is compression, and wherein the second data stream is compressed in the storage layer.
17. The non-transitory computer readable medium of claim 15 , wherein the data augmentation group is encryption, and wherein the second data stream is encrypted in the storage layer.
18. The non-transitory computer readable medium of claim 15 , wherein processing file system operations received from the file system client includes processing a write operation, wherein processing the write operation includes:
writing file data associated with the write operation to the cache overlay; and
marking the chunk state in the cache overlay inode as dirty for each chunk of file data implicated by the write operation.
19. The non-transitory computer readable medium of claim 15 , wherein processing file system operations includes processing a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.
20. The non-transitory computer readable medium of claim 15 , wherein processing file system operations includes processing a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
augmenting the set of chunks based on the data augmentation group associated with the storage layer inode of the file;
generating a set of augmentation metadata based on the augmenting;
storing the augmented set of chunks and the set of augmentation metadata in the storage layer based on the storage layer inode of the file; and
changing the chunk state of the set of chunks to clean.