Techniques for fast IO and low memory consumption while using erasure codes
Methods and systems for improving the read and write performance of a distributed file system while limiting memory usage are described. The type of error correcting scheme applied to data, the partitioning of the data into data chunks, and the sizes of data slices within each of the data chunks used for storing electronic files within the distributed file system may be dynamically adjusted over time to optimize for fast IO performance while limiting memory usage (e.g., requiring less than 256 MB of RAM to generate and store code blocks). The file size of an electronic file to be stored, the amount of available memory for generating code blocks, and the amount of available disk space to store the electronic file may be used to set the data sizes of the data slices and the type of erasure code applied to data blocks associated with the data slices.
1. A method for operating a data management system, comprising:
partitioning an electronic file into a plurality of stripes, the plurality of stripes includes a first stripe;
determining a first data size for the first stripe;
determining an amount of available memory for storing data blocks and code blocks associated with a first wedge of the first stripe;
determining a first data slice length for data slices within the first wedge of the first stripe based on the first data size for the first stripe and the amount of available memory, wherein a different size of the first data slice length is determined based on a location of the electronic file, wherein the electronic file comprises a database file and a virtual disk file;
identifying a first set of data slices corresponding with the first wedge based on the first data slice length, the first set of data slices includes a first set of data blocks and a first set of code blocks;
writing the first set of data blocks and the first set of code blocks to a plurality of disks such that each disk of the plurality of disks stores one of the first set of data blocks or one of the first set of code blocks; and
determining an amount of available disk space for the plurality of disks, wherein the determining a first error correcting scheme to apply to the first set of data blocks includes determining the first error correcting scheme based on the amount of available disk space for the plurality of disks.
2. The method of claim 1 , wherein:
the determining the first data slice length for the data slices within the first wedge includes detecting that the amount of available memory is less than a threshold memory size and decreasing the first data slice length in response to detecting that the amount of available memory is less than the threshold memory size.
3. The method of claim 1 , further comprising:
determining an amount of available disk space within the plurality of disks; and
the determining the first data slice length for the data slices within the first wedge includes determining the first data slice length for the data slices within the first wedge based on the amount of available disk space within the plurality of disks.
4. The method of claim 3 , wherein:
the determining the first data slice length for the data slices within the first wedge includes detecting that the amount of available disk space is less than a threshold disk space size and decreasing the first data slice length in response to detecting that the amount of available disk space is less than the threshold disk space size.
5. The method of claim 1 , further comprising:
determining a second data size different from the first data size for a second stripe of the plurality of stripes;
determining a second data slice length different from the first data slice length for data slices within a second wedge of the second stripe based on the second data size for the second stripe;
identifying a second set of data slices corresponding with the second wedge based on the second data slice length, the second set of data slices includes a second set of data blocks and a second set of code blocks; and
writing the second set of data blocks and the second set of code blocks to the plurality of disks.
6. The method of claim 5 , wherein:
the first data slice length corresponds with 64 KB of data; and
the second data slice length corresponds with 128 KB of data.
7. The method of claim 1 , wherein:
the plurality of disks includes two or more hard disk drives.
8. The method of claim 1 , further comprising:
determining the first error correcting scheme to apply to the first set of data blocks based on the first data size for the first stripe and the amount of available memory; and
generating the first set of code blocks using the first of data blocks and the first error correcting scheme.
9. The method of claim 8 , wherein:
the first error correcting scheme comprises a Reed-Solomon encoding.
10. A data management system, comprising:
a plurality of disks; and
one or more processors configured to partition an electronic file into a plurality of stripes, the plurality of stripes includes a first stripe, the one or more processors configured to determine a first data size for the first stripe and determine an amount of available memory for storing data blocks and code blocks associated with a first wedge of the first stripe, the one or more processors configured to determine a first data slice length for data slices within the first wedge of the first stripe based on the first data size for the first stripe and the amount of available memory. wherein a different size of the first data slice length is determined based on a location of the electronic file, wherein the electronic file comprises a database file and a virtual disk file, the one or more processors configured to identify a first set of data slices corresponding with the first wedge based on the first data slice length, the first set of data slices includes a first set of data blocks and a first set of code blocks, the one or more processors configured to write the first set of data blocks and the first set of code blocks to a plurality of disks such that each disk of the plurality of disks stores one of the first set of data blocks or one of the first set of code blocks, the one or more processors configured to determine an amount of available disk space for the plurality of disks, wherein the determining a first error correcting scheme to apply to the first set of data blocks includes determining the first error correcting scheme based on the amount of available disk space for the plurality of disks.
11. The data management system of claim 10 , wherein:
the one or more processors configured to detect that the amount of available memory is less than a threshold memory size and decrease the first data slice length in response to detection that the amount of available memory is less than the threshold memory size.
12. The data management system of claim 10 , wherein:
the one or more processors configured to determine an amount of available disk space within the plurality of disks and determine the first data slice length for the data slices within the first wedge based on the amount of available disk space within the plurality of disks.
13. The data management system of claim 12 , wherein:
the one or more processors configured to detect that the amount of available disk space is less than a threshold disk space size and decrease the first data slice length in response to detection that the amount of available disk space is less than the threshold disk space size.
14. The data management system of claim 10 , wherein:
the one or more processors configured to determine a second data size different from the first data size for a second stripe of the plurality of stripes and determine a second data slice length different from the first data slice length for data slices within a second wedge of the second stripe based on the second data size for the second stripe, the one or more processors identify a second set of data slices corresponding with the second wedge based on the second data slice length, the second set of data slices includes a second set of data blocks and a second set of code blocks, the one or more processors configured to write the second set of data blocks and the second set of code blocks to the plurality of disks.
15. The data management system of claim 10 , wherein:
the first data slice length corresponds with 64 KB of data; and
the second data slice length corresponds with 128 KB of data.
16. The data management system of claim 10 , wherein:
the plurality of disks includes two or more hard disk drives.
17. A method for operating a data management system, comprising:
partitioning an electronic file into a plurality of stripes, the plurality of stripes includes a first stripe;
determining a first data size for the first stripe;
determining an amount of available memory for storing data blocks and code blocks associated with a first wedge of the first stripe;
determining a first error correcting scheme to apply to data blocks of the first wedge of the first stripe based on the first data size for the first stripe and the amount of available memory, wherein the electronic file comprises a database file;
identifying a first set of data blocks associated with the first wedge of the first stripe;
generating a first set of code blocks using the first of data blocks and the first error correcting scheme; and
writing the first set of data blocks and the first of code blocks to a plurality of disks such that each disk of the plurality of disks stores one of the first set of data blocks or one of the first set of code blocks.
18. The method of claim 17 , further comprising:
determining a second data size different from the first data size for a second stripe of the plurality of stripes;
determining a second error correcting scheme to apply to data blocks corresponding with a second wedge of the second stripe based on the second data size for the second stripe and the amount of available memory, wherein the electronic file comprises a virtual disk file;
identifying a second set of data blocks associated with the second wedge of the second stripe;
generating a second set of code blocks using the second set of data blocks and the second error correcting scheme; and
concurrently writing the second set of data blocks and the second set of code blocks to the plurality of disks.
19. The method of claim 18 , wherein:
the first error correcting scheme comprises a first Reed Solomon encoding with four data blocks and two code blocks; and
the second error correcting scheme comprises a second Reed Solomon encoding with six data blocks and three code blocks.