Remote durable logging for journaling file systems
A journaling file system may implement remote durable logging. Updates to a file system may be received, and log records describing the updates may be stored in a locally-accessible file system change log. The update may then be acknowledged as committed. The log records may then be sent to be stored in a network-based data store in a remote version of the file system change log. Once it may be determined that the log records are stored in the remote version, storage space for the log records in the local file system change log may be reclaimed. Various types of restoration and duplication techniques may be implemented based on the remote version of the change log to restore a file system at an originating device or to duplicate the file system at a different device.
1 . A method, comprising:
performing, by one or more computing devices that implement a file system hosting service at a cloud-based infrastructure provider network:
registering a file system with the file system hosting service;
creating a duplicate file system of the file system at the file system hosting service, wherein the duplicate file system is created based on a snapshot of the file system at a remote location and a remote version of a file system change log at the remote location;
provisioning, based on expected I/O operations per second (IOPS) specified in a corresponding request, one or more virtual machine-based read nodes for the duplicate file system, wherein the one or more read nodes are virtual machine instances managed by a virtual computing service of the cloud-based infrastructure provider network, provisioned with computational capacity, memory size, and software stack specified for the virtual machines instances by the virtual computing service of the cloud-based infrastructure provider network based at least in part on the expected IOPS specified in the corresponding request;
making the duplicate file system available for read access requests via the one or more read nodes; and
determining that additional log records have been added to the remote version of the file system change log, and in response:
updating the duplicate file system based on the additional log records.
2 . The method of claim 1 , wherein:
the cloud-based infrastructure provider network provides multi-tenant storage for a plurality of clients accessible over a public network;
the file system is associated with a client of the cloud-based infrastructure provider network and located at a first location of the client; and
the duplicate file system is located at a second location of the cloud-based infrastructure provider network remote from the first location.
3 . The method of claim 2 , wherein:
the duplicate file system is used to maintain a second duplicate file system; and
the second duplicate file system is located at a third location of the client remote from the first and second locations.
4 . The method of claim 1 , wherein:
the duplicate file system is configured to handle a specified number of input/output operations per second (IOPS).
5 . The method of claim 4 , wherein:
the one or more read nodes are part of a read pool provisioned for the duplicate file system; and
the method further comprises selecting a size of the read pool is based at least in part on the IOPS configured for the duplicate file system.
6 . The method of claim 5 , further comprising:
dynamically scaling the size of the read pool based at least in part on an expected workload of the read pool.
7 . The method of claim 1 , further comprising:
provisioning one or more storage resources to store the duplicate file system in the cloud-based infrastructure provider network, wherein the storage resources include one or more solid state drive (SSD) devices.
8 . The method of claim 1 , wherein:
the cloud-based infrastructure provider network implements user authentication to control access to the duplicate file system.
9 . The method of claim 1 , further comprising:
tracking usage of the duplicate file system by a client; and
generating a report of the usage in response to a query from the client.
10 . The method of claim 9 , wherein:
the cloud-based infrastructure provider network provides a configuration interface for configuring the duplicate file system; and
the configuration interface comprises an application programing interface (API).
11 . A system, comprising:
one or more computing devices that implement a file system hosting service at cloud-based infrastructure provider network, configured to:
register a file system with the file system hosting service;
create a duplicate file system of the file system at the file system hosting service, wherein the duplicate file system is created based on a snapshot of the file system at a remote location and a remote version of a file system change log at the remote location;
provision, based on expected I/O operations per second (IOPS) specified in a corresponding request, one or more virtual machine-based read nodes for the duplicate file system, wherein the one or more read nodes are virtual machine instances managed by a virtual computing service of the cloud-based infrastructure provider network, provisioned with computational capacity, memory size, and software stack specified for the virtual machines instances by the virtual computing service of the cloud-based infrastructure provider network based at least in part on the expected IOPS specified in the corresponding request;
make the duplicate file system available for read access requests via the one or more read nodes; and
determine that additional log records have been added to the remote version of the file system change log, and in response:
update the duplicate file system based on the additional log records.
12 . The system of claim 11 , wherein:
the one or more read nodes are part of a read pool provisioned for the duplicate file system; and
a size of the read pool is selected based at least in part on an amount of input/output operations per second (IOPS) configured for the duplicate file system.
13 . The system of claim 12 , wherein the file system hosting service is configured to:
dynamically scale the size of the read pool based at least in part on an expected workload of the read pool.
14 . The system of claim 12 , wherein the file system hosting service is configured to:
dynamically scale the size of the read pool based at least in part on a workload of the read pool.
15 . The system of claim 11 , wherein to update the duplicate file system based on the additional log records, the file system hosting service is configured to:
receive the additional log records from the remote location;
reconcile the additional log records with a local log of the duplicate file system stored at the cloud-based infrastructure provider network; and
after the reconciliation, apply the reconciled the additional log records to the duplicate file system.
16 . The system of claim 11 , wherein the additional log records indicate modifications to metadata of the file system including one or more of:
a deletion of a file;
a move of a file;
a modification of a directory structure of the file system; and
a modification of permissions of a file.
17 . The system of claim 11 , wherein the additional log records indicate changes to contents of a file.
18 . The system of claim 11 , wherein the file system hosting service is configured to perform automated backups of the duplicate file system.
19 . The system of claim 11 , wherein the file system hosting service is configured to provide read-only access to the duplicate file system.
20 . The system of claim 11 , wherein:
the cloud-based infrastructure provider network provides a configuration interface for configuring the duplicate file system; and
the configuration interface is a web-based interface provided according to Hypertext Transfer Protocol (HTTP).