Transparent hybrid data storage
View Patent ↗A system, method, and apparatus are provided for performing a transparent hybrid data storage scheme in which data are stored as blocks distributed among one or more flash-based storage devices (e.g., solid state drives) and one or more magnetic storage devices (e.g., magnetic disk drives). Files larger than a given size (e.g., 1 MB) are segmented into blocks of that size and stored on one or more devices; blocks of one file may be stored on devices of different types. Periodically, a utility function calculates utility values for each of some or all stored blocks based on frequency of access to the block, frequency of access of a particular type (e.g., random, sequential), a preference regarding where to store the block or the corresponding file, and/or other factors. Blocks having the highest utility values are subject to migration between devices of different types and/or the same type (e.g., for load-balancing).
1. A method comprising:
partitioning multiple files into fixed-sized blocks;
receiving, from a user of a first file among the multiple files, a user preference regarding a type of storage device on which to store the first file;
load-balancing a first portion of the blocks among one or more flash-based storage devices based on one or more of:
remaining lifetimes of the one or more flash-based storage devices; and
predicted future access to the first portion of blocks;
storing a second portion of the blocks on one or more magnetic storage devices;
periodically executing a utility function to calculate utility values of each of the blocks; and
migrating a plurality of the blocks between the flash-based storage devices and the magnetic storage devices based on the utility values.
2. The method of claim 1 , further comprising, for each of the plurality of blocks:
monitoring a number of accesses to the block;
monitoring a number of accesses to the block that are of a particular type; and
identifying a user preference regarding storage of the block on a flash-based storage device or a magnetic storage device.
3. The method of claim 2 , wherein executing the utility function for a given block among the plurality of blocks comprises:
combining the number of accesses to the given block, the number of accesses to the block that are of a particular type, and the user preference;
wherein the particular type is one of random access and sequential access.
4. The method of claim 3 , wherein the number of accesses to the given block and the number of accesses to the block that are of a particular type are modified by a decay function prior to said combining.
5. The method of claim 1 , wherein at least one block of a first file is stored on a flash-based storage device and at least one other block of the first file is stored on a magnetic storage device.
6. An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
partition multiple files into fixed-sized blocks;
receive, from a user of a first file among the multiple files, a user preference regarding a type of storage device on which to store the first file;
load-balance a first portion of the blocks among one or more flash-based storage devices based on one or more of:
remaining lifetimes of the one or more flash-based storage devices; and
predicted future access to the first portion of blocks;
store a second portion of the blocks on one or more magnetic storage devices;
periodically execute a utility function to calculate utility values of each of the blocks; and
migrate a plurality of the blocks between the flash-based storage devices and the magnetic storage devices based on the utility values.
7. The apparatus of claim 6 , the memory further storing instructions that, when executed by the one or more processors, cause the apparatus to, for each of the plurality of blocks:
monitor a number of accesses to the block;
monitor a number of accesses to the block that are of a particular type; and
identify a user preference regarding storage of the block on a flash-based storage device or a magnetic storage device.
8. The apparatus of claim 7 , wherein executing the utility function for a given block among the plurality of blocks comprises:
combining the number of accesses to the given block, the number of accesses to the block that are of a particular type, and the user preference;
wherein the particular type is one of random access and sequential access.
9. The apparatus of claim 8 , wherein the number of accesses to the given block and the number of accesses to the block that are of a particular type are modified by a decay function prior to said combining.
10. The apparatus of claim 6 , wherein at least one block of a first file is stored on a flash-based storage device and at least one other block of the first file is stored on a magnetic storage device.
11. A system, comprising:
at least one processor;
at least one flash-based storage device;
at least one magnetic storage device;
a storage allocation module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
partition multiple files into fixed-sized blocks;
receive, from a user of a first file among the multiple files, a user preference regarding a type of storage device on which to store the first file;
load-balance a first portion of the blocks among one or more of the flash-based storage devices based on one or more of:
remaining lifetimes of the one or more flash-based storage devices; and
predicted future access to the first portion of blocks; and
store a second portion of the blocks on one or more of the magnetic storage devices;
a utility module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to periodically execute a utility function to calculate utility values of each of the blocks; and
a migration module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to migrate a plurality of the blocks between the flash-based storage devices and the magnetic storage devices based on the utility values.
12. The system of claim 11 , further comprising:
a monitor module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to, for each of the plurality of blocks:
monitor a number of accesses to the block; and
monitor a number of accesses to the block that are of a particular type; and
a preference module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to, for each of the plurality of blocks:
identify a user preference regarding storage of the block on a flash-based storage device or a magnetic storage device.
13. The system of claim 12 , wherein executing the utility function for a given block among the plurality of blocks comprises:
combining the number of accesses to the given block, the number of accesses to the block that are of a particular type, and the user preference;
wherein the particular type is one of random access and sequential access.
14. The system of claim 13 , wherein the number of accesses to the given block and the number of accesses to the block that are of a particular type are modified by a decay function prior to said combining.
15. The system of claim 11 , wherein at least one block of a first file is stored on a flash-based storage device and at least one other block of the first file is stored on a magnetic storage device.