Consistent file system semantics with cloud object storage
Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.
1 . A non-transitory computer-readable medium including instructions that when executed by one or more processors, cause a system including the one or more processors to perform operations including:
receiving a cloud storage object from a cloud object store;
staging the cloud storage object in a first cache device of a plurality of cache devices, wherein the first cache device is classified within a first stage of cache devices, and a second cache device is classified within a second stage of cache devices;
restaging the cloud storage object from the first cache device to the second cache device,
wherein restaging the cloud storage object comprises:
determining a transition criteria for restaging the cloud storage object, wherein determining the transition criteria comprises performing one or both of: (i) determining a plurality of recency attributes for the plurality of other cloud storage objects stored within the second stage of cache devices, and determining a recency threshold based at least in part on the plurality of recency attributes, and (ii) determining a plurality of frequency attributes for the plurality of other cloud storage objects stored within the second stage of cache devices, and determining a frequency threshold based at least in part on the plurality of frequency attributes,
determining a recency attribute and/or a frequency attribute of the cloud storage object stored in the first cache device, and
performing one or both of (i) comparing the recency attribute of the cloud storage object with the recency threshold and (ii) comparing the frequency attribute of the cloud storage object with the frequency threshold;
and based at least in part on comparing the recency attribute of the cloud storage object with the recency threshold and/or comparing the frequency attribute of the cloud storage object with the frequency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
2 . The non-transitory computer-readable medium of claim 1 , wherein a first frequency attribute of the plurality of frequency attributes is indicative of a number of accesses of a first other cloud storage object of the plurality of other cloud storage objects.
3 . The non-transitory computer-readable medium of claim 1 , wherein restaging the cloud storage object further comprises one or both of (i) in response to the recency attribute of the cloud storage object being higher than the recency threshold and (ii) in response to the frequency attribute of the cloud storage object being higher than the frequency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
4 . The non-transitory computer-readable medium of claim 3 , wherein
restaging the cloud storage object further comprises: in response to the recency attribute of the cloud storage object being higher than the recency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
5 . The non-transitory computer-readable medium of claim 1 , wherein restaging the cloud storage object further comprises: in response to the frequency attribute of the cloud storage object being higher than the frequency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
6 . The non-transitory computer-readable medium of claim 1 , wherein the frequency threshold is a lowest frequency attribute of the plurality of frequency attributes for the plurality of other cloud storage objects.
7 . The non-transitory computer-readable medium of claim 5 , wherein:
the second stage of cache devices are configured to store first one or more cloud storage objects that are accessed with a higher frequency; and
the first stage of cache devices are configured to store second one or more cloud storage objects that are accessed with a lower frequency.
8 . The non-transitory computer-readable medium of claim 5 , wherein a first performance-capability characteristics of the first stage of cache devices is different from a second performance-capability characteristics of the second stage of cache devices.
9 . The non-transitory computer-readable medium of claim 1 , wherein a first recency attribute of the plurality of recency attributes is indicative of a last access time of a first other cloud storage object of the plurality of other cloud storage objects.
10 . The non-transitory computer-readable medium of claim 1 , wherein:
the recency threshold is a function of a lowest value of the plurality of recency attributes for the plurality of other cloud storage objects.
11 . The non-transitory computer-readable medium of claim 1 , wherein receiving the cloud storage object from the cloud object store comprises:
receiving a request to perform a transaction with respect to a file;
translating the request into an I/O request to perform an I/O operation;
determining that none of the plurality of cache devices local to a block storage system includes at least one data object corresponding to the I/O operation;
generating an object interface request based at least in part on the I/O request;
transmitting the object interface request to the cloud object store, wherein the object interface request corresponds to a read operation with respect to an instance of the at least one data object stored in the cloud object store;
receiving the cloud storage object from the cloud object store, wherein the at least one cloud storage object corresponds to the instance of the at least one data object.
12 . The non-transitory computer-readable medium of claim 1 , wherein the recency threshold and/or the frequency threshold are dynamic thresholds that change in response to changes in the plurality of recency attributes for the plurality of other cloud storage objects and/or changes in the plurality of frequency attributes for the plurality of other cloud storage objects.
13 . A method comprising:
receiving a cloud storage object from a cloud object store;
staging the cloud storage object in a first cache device of a plurality of cache devices, wherein the first cache device is classified within a first stage of cache devices, and a second cache device is classified within a second stage of cache devices;
restaging the cloud storage object from the first cache device to the second cache device, wherein restaging the cloud storage object comprises:
determining a transition criteria for restaging the cloud storage object, wherein determining the transition criteria comprises performing one or both of: (i) determining a plurality of recency attributes for the plurality of other cloud storage objects stored within the second stage of cache devices, and determining a recency threshold based at least in part on the plurality of recency attributes, and (ii) determining a plurality of frequency attributes for the plurality of other cloud storage objects stored within the second stage of cache devices, and determining a frequency threshold based at least in part on the plurality of frequency attributes,
determining a recency attribute and/or a frequency attribute of the cloud storage object stored in the first cache device, and
performing one or both of (i) comparing the recency attribute of the cloud storage object with the recency threshold and (ii) comparing the frequency attribute of the cloud storage object with the frequency threshold;
and based at least in part on comparing the recency attribute of the cloud storage object with the recency threshold and/or comparing the frequency attribute of the cloud storage object with the frequency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
14 . The method of claim 13 , wherein a first frequency attribute of the plurality of frequency attributes is indicative of a number of accesses of a first other cloud storage object of the plurality of other cloud storage objects.
15 . The method of claim 13 , wherein restaging the cloud storage object further comprises one or both of (i) in response to the recency attribute of the cloud storage object being higher than the recency threshold and (ii) in response to the frequency attribute of the cloud storage object being higher than the frequency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
16 . The method of claim 13 , wherein a first recency attribute of the plurality of recency attributes is indicative of a last access time of a first other cloud storage object of the plurality of other cloud storage objects.
17 . The method of claim 13 , wherein:
the recency threshold is a function of a lowest value of the plurality of recency attributes defined for the plurality of other cloud storage objects.
18 . A system comprising:
one or more data processors; and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of actions including:
receiving a cloud storage object from a cloud object store;
staging the cloud storage object in a first cache device of a plurality of cache devices, wherein the first cache device is classified within a first stage of cache devices, and a second cache device is classified within a second stage of cache devices;
restaging the cloud storage object from the first cache device to the second cache device, wherein restaging the cloud storage object comprises:
determining a transition criteria for restaging the cloud storage object, wherein determining the transition criteria comprises performing one or both of: (i) determining a plurality of recency attributes for the plurality of other cloud storage objects stored within the second stage of cache devices, and determining a recency threshold based at least in part on the plurality of recency attributes, and (ii) determining a plurality of frequency attributes for the plurality of other cloud storage objects stored within the second stage of cache devices, and determining a frequency threshold based at least in part on the plurality of frequency attributes,
determining a recency attribute and/or a frequency attribute of the cloud storage object stored in the first cache device, and
performing one or both of (i) comparing the recency attribute of the cloud storage object with the recency threshold and (ii) comparing the frequency attribute of the cloud storage object with the frequency threshold;
and based at least in part on comparing the recency attribute of the cloud storage object with the recency threshold and/or comparing the frequency attribute of the cloud storage object with the frequency threshold, restaging the cloud storage object from the first cache device of the plurality of cache devices to the second cache device of the plurality of cache devices.
19 . The system of claim 18 , wherein a first frequency attribute of the plurality of frequency attributes is indicative of a number of accesses of a first other cloud storage object of the plurality of other cloud storage objects.
20 . The system of claim 19 , wherein a first recency attribute of the plurality of recency attributes is indicative of a last access time of a first other cloud storage object of the plurality of other cloud storage objects.