File layer to block layer communication for block organization in storage
A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and (3) organizing the data blocks within the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.
1. A method, performed by a block-storage server, of storing data, the method comprising:
receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server;
receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and
organizing the data blocks within the persistent block storage based, at least in part, on the received metadata;
wherein the metadata for each data block indicates a file to which that data block belongs;
wherein organizing the data blocks includes storing data blocks belonging to respective files in respective locations of the persistent block storage together with other data blocks belonging to the respective files; and
wherein storing data blocks belonging to respective particular files in respective locations of the persistent block storage includes segregating a set of data blocks belonging to different files in respective stripes of the persistent storage, including arranging data blocks belonging to a first file in a first stripe of the persistent block storage and arranging data blocks belonging to a second file in a second stripe of the persistent block storage.
2. The method of claim 1 wherein storing data blocks belonging to respective particular files in respective locations of the persistent block storage includes:
aggregating multiple data blocks belonging to a particular file into a data extent;
compressing the data extent, yielding a compressed extent; and
storing the compressed extent in the persistent block storage.
3. The method of claim 1 wherein arranging the data blocks belonging to the first file in the first stripe includes:
aggregating multiple data blocks belonging to the first file into a data extent;
compressing the data extent, yielding a compressed extent; and
storing the compressed extent in the first stripe of the persistent block storage.
4. The method of claim 3 wherein arranging the data blocks belonging to the first file in the first stripe further includes, prior to aggregating the multiple data blocks belonging to the first file into the data extent, determining that the multiple data blocks belonging to the first file are to be compressed.
5. The method of claim 4 wherein storing data blocks belonging to respective particular files in respective locations of the persistent block storage further includes:
determining that blocks belonging to a third file are not to be compressed; and
storing a plurality of blocks belonging to the third file that are in the first stripe directly within the first stripe without aggregating or compressing the plurality of blocks belonging to the third file that are in the first stripe.
6. The method of claim 1 wherein the metadata includes file identifiers that identify files uniquely with respect to all filesystems managed by file servers that are configured to send data blocks to the block-storage server.
7. The method of claim 6 wherein receiving the metadata includes receiving a file identifier together with each of the data blocks.
8. The method of claim 6 wherein the metadata includes a single file identifier embedded within a write command received from the remote file server, the write command indicating a set of multiple data blocks from the remote file server that belong to a same file.
9. The method of claim 6 wherein each file identifier includes (i) a filesystem identifier that uniquely identifies a filesystem and (ii) an inode identifier that uniquely identifies a file within that filesystem.
10. The method of claim 1 wherein receiving the metadata includes:
receiving a reference to a particular inode stored within the persistent block storage, the particular inode provided for a particular file managed by the remote file server; and
identifying received data blocks that belong to the particular file based at least in part on traversing a block tree associated with the particular inode.
11. A computer program product comprising a non-transitory computer-readable storage medium storing instructions, which, when executed by a block-storage server, cause the block-storage server to:
receive, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server;
receive, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and
organize the data blocks within the persistent block storage based, at least in part, on the received metadata;
wherein the metadata for each data block indicates a file to which that data block belongs;
wherein organizing the data blocks includes storing data blocks belonging to respective files in respective locations of the persistent block storage together with other data blocks belonging to the respective files; and
wherein storing data blocks belonging to respective particular files in respective locations of the persistent block storage includes segregating a set of data blocks belonging to different files in respective stripes of the persistent storage, including arranging data blocks belonging to a first file in a first stripe of the persistent block storage and arranging data blocks belonging to a second file in a second stripe of the persistent block storage.
12. The computer program product of claim 11 ,
wherein the metadata includes file identifiers that identify files uniquely with respect to all filesystems managed by file servers that are configured to send data blocks to the block-storage server; and
wherein each file identifier includes (i) a filesystem identifier that uniquely identifies a filesystem and (ii) an inode identifier that uniquely identifies a file within that filesystem.
13. The computer program product of claim 11 ,
wherein receiving the metadata includes:
receiving a reference to a particular inode stored within the persistent block storage, the particular inode provided for a particular file managed by the remote file server; and
identifying received data blocks that belong to the particular file based at least in part on traversing a block tree associated with the particular inode.
14. The computer program product of claim 11 wherein arranging the data blocks belonging to the first file in the first stripe includes:
aggregating multiple data blocks belonging to the first file into a data extent;
compressing the data extent, yielding a compressed extent; and
storing the compressed extent in the first stripe of the persistent block storage.
15. The computer program product of claim 14 wherein arranging the data blocks belonging to the first file in the first stripe further includes, prior to aggregating the multiple data blocks belonging to the first file into the data extent, determining that the multiple data blocks belonging to the first file are to be compressed.
16. An apparatus comprising:
persistent block storage managed by the apparatus; and
a controller coupled to memory configured to:
receive, from a remote file server, data blocks to be written to the persistent block storage;
receive, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and
organize the data blocks within the persistent block storage based, at least in part, on the received metadata;
wherein the metadata for each data block indicates a file to which that data block belongs;
wherein organizing the data blocks includes storing data blocks belonging to respective files in respective locations of the persistent block storage together with other data blocks belonging to the respective files; and
wherein storing data blocks belonging to respective particular files in respective locations of the persistent block storage includes segregating a set of data blocks belonging to different files in respective stripes of the persistent storage, including arranging data blocks belonging to a first file in a first stripe of the persistent block storage and arranging data blocks belonging to a second file in a second stripe of the persistent block storage.
17. The apparatus of claim 16 ,
wherein the metadata includes file identifiers that identify files uniquely with respect to all filesystems managed by file servers that are configured to send data blocks to the block-storage server; and
wherein each file identifier includes (i) a filesystem identifier that uniquely identifies a filesystem and (ii) an inode identifier that uniquely identifies a file within that filesystem.
18. The apparatus of claim 16 ,
wherein receiving the metadata includes:
receiving a reference to a particular inode stored within the persistent block storage, the particular inode provided for a particular file managed by the remote file server; and
identifying received data blocks that belong to the particular file based at least in part on traversing a block tree associated with the particular inode.
19. The apparatus of claim 16 wherein arranging the data blocks belonging to the first file in the first stripe includes:
aggregating multiple data blocks belonging to the first file into a data extent;
compressing the data extent, yielding a compressed extent; and
storing the compressed extent in the first stripe of the persistent block storage.
20. The apparatus of claim 19 wherein arranging the data blocks belonging to the first file in the first stripe further includes, prior to aggregating the multiple data blocks belonging to the first file into the data extent, determining that the multiple data blocks belonging to the first file are to be compressed.