Apparatus and method for managing storage of data blocks
A data block storage management capability is presented. A cloud file system management capability manages storage of data blocks of a file system across multiple cloud storage services (e.g., including determining, for each data block to be stored, a storage location and a storage duration for the data block). A cloud file system management capability manages movement of data blocks of a file system between storage volumes of cloud storage services. A cloud file system management capability provides a probabilistic eviction scheme for evicting data blocks from storage volumes of cloud storage services in advance of storage deadlines by which the data blocks are to be removed from the storage volumes. A cloud file system management capability enables dynamic adaptation of the storage volume sizes of the storage volumes of the cloud storage services.
1. An apparatus, comprising:
a processor and a memory communicatively connected to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform a method, the method comprising:
determining a storage location and a storage duration for a data block using cost information associated with a plurality of cloud storage services available for storing data blocks of a file system, the storage location specifying one of the plurality of cloud storage services in which the data block is to be stored, the storage duration specifying a length of time for which the data block is to be stored in the storage location; and
analyzing storage of the data block at the storage location, based on an access pattern of the data block and the storage duration determined for the data block, to determine whether to remove the data block from the storage location by performing the steps of:
monitoring access requests for the data block; and
when an amount of time equal to the storage duration has elapsed without an intervening access request for the data block, initiating removal of the data block from the storage location, wherein:
the cost information comprises at least one of a storage cost and an access cost associated with each of the plurality of cloud storage services;
the plurality of cloud storage services include a first cloud storage service having a lower storage cost and a higher access cost than at least a second cloud storage service;
the storage location specifies the second cloud storage service; and
the processor is configured to:
determine a cost of accessing data from the first cloud storage service (f 1 );
determine a cost of storing data in the second cloud storage service (m 2 ); and
compute the storage duration T for the data block based on T=(f 1 )/(m 2 ).
2. The apparatus of claim 1 , wherein the cost information further comprises a bandwidth cost associated with the cloud storage service.
3. The apparatus of claim 1 , wherein, for at least one of the plurality of cloud storage services, the cost information associated with the cloud storage service is determined using at least one of a fixed pricing model and a dynamic pricing model.
4. The apparatus of claim 1 , wherein, to analyze storage of the data block at the storage location for determining whether to transfer the data block from the storage location, the processor is further configured to:
determine, for the data block, whether to evict the data block from the storage location prior to removal of the data block from the storage location based on the storage duration.
5. The apparatus of claim 1 , wherein, to analyze storage of the data block at the storage location for determining whether to transfer the data block from the storage location, the processor is further configured to:
when a data block is stored in the storage location or accessed from the storage location, compute a data block eviction time for the data block using a probability distribution; and
initiate eviction of the data block from the storage location when the data block eviction time elapses without a subsequent access to the data block.
6. The apparatus of claim 1 , wherein the file is stored in a storage volume of the one of the plurality of cloud file services indicated by the file storage location, wherein the processor is configured to:
initiate a resizing event for resizing the storage volume.
7. The apparatus of claim 6 , wherein the size of the storage volume at a current resizing event is S, wherein the processor is configured to:
after the current resizing event, monitor a number of data blocks replaced in S before their respective block eviction times (B replace ) and monitor a number of blocks evicted from S (B evict ); and
at a next resizing event, determine a new size of the storage volume using S+B replace −B evict .
8. The apparatus of claim 7 , wherein the processor is configured to:
determine a resizing granularity of the one of the plurality of cloud storage services indicated by the file storage location; and
round the determined new size of the storage volume to a nearest multiple of the resizing granularity.
9. The apparatus of claim 6 , wherein the resizing event comprises:
an increase resizing events configured to attempt to increase the size of the storage volume; or
a decrease resizing events configured to attempt to decrease the size of the storage volume.
10. The apparatus of claim 1 , wherein the file is stored in a storage volume of the one of the plurality of cloud file services indicated by the file storage location, wherein the processor is configured to:
periodically initiate resizing events for resizing the storage volume.
11. The apparatus of claim 10 , wherein the resizing events comprise:
increase resizing events configured to attempt to increase the size of the storage volume; and
decrease resizing events configured to attempt to decrease the size of the storage volume.
12. The apparatus of claim 11 , wherein a periodic interval associated with the increase resizing events is less than a periodic interval associated with the decrease resizing events.
13. The apparatus of claim 12 , wherein the periodic interval associated with the decrease resizing events is determined based on a minimum storage time for which a customer may pay for storage of data in the one of the plurality of cloud storage services.
14. The apparatus of claim 1 , wherein the processor is configured to:
when a determination is made to remove or evict the data block from the storage location, determine a new file storage location for the data block.
15. The apparatus of claim 1 , wherein the processor is configured to determine a new file storage location for the data block based on the cost information associated with the plurality of cloud storage services available for storing data blocks of the file system.
16. A method, comprising:
using a processor and a memory for:
determining a storage location and a storage duration for a data block using cost information associated with a plurality of cloud storage services available for storing data blocks of a file system, the storage location specifying one of the plurality of cloud storage services in which the data block is to be stored, the storage duration specifying a length of time for which the data block is to be stored in the storage location; and
analyzing storage of the data block at the storage location, based on an access pattern of the data block and the storage duration determined for the data block, for determining whether to remove the data block from the storage location by performing the steps of:
monitoring access requests for the data block; and
when an amount of time equal to the storage duration has elapsed without an intervening access request for the data block, initiating removal of the data block from the storage location, wherein:
the cost information comprises at least one of a storage cost and an access cost associated with each of the plurality of cloud storage services;
the plurality of cloud storage services include a first cloud storage service having a lower storage cost and a higher access cost than at least a second cloud storage service;
the storage location specifies the second cloud storage service; and
the processor is configured to:
determine a cost of accessing data from the first cloud storage service (f 1 );
determine a cost of storing data in the second cloud storage service (m 2 ); and
compute the storage duration T for the data block based on T=(f 1 )/(m 2 ).
17. A non-transitory computer-readable storage medium storing instructions which, when executed by a computer, cause the computer to perform a method, the method comprising:
determining a storage location and a storage duration for a data block using cost information associated with a plurality of cloud storage services available for storing data blocks of a file system, the storage location specifying one of the plurality of cloud storage services in which the data block is to be stored, the storage duration specifying a length of time for which the data block is to be stored in the storage location; and analyzing storage of the data block at the storage location, based on an access pattern of the data block and the storage duration determined for the data block, for determining whether to remove the data block from the storage location by performing the steps of:
monitoring access requests for the data block; and
when an amount of time equal to the storage duration has elapsed without an intervening access request for the data block, initiating removal of the data block from the storage location, wherein:
the cost information comprises at least one of a storage cost and an access cost associated with each of the plurality of cloud storage services;
the plurality of cloud storage services include a first cloud storage service having a lower storage cost and a higher access cost than at least a second cloud storage service;
the storage location specifies the second cloud storage service; and
the processor is configured to:
determine a cost of accessing data from the first cloud storage service (f 1 );
determine a cost of storing data in the second cloud storage service (m 2 ); and
compute the storage duration T for the data block based on T=(f 1 )/(m 2 ).