Chunk allocation
Methods and systems for identifying a set of disks within a cluster and then storing a plurality of data chunks into the set of disks such that the placement of the plurality of data chunks within the cluster optimizes failure tolerance and storage system performance for the cluster are described. The plurality of data chunks may be generated using replication of data (e.g., n-way mirroring) or application of erasure coding to the data (e.g., using a Reed-Solomon code or a Low-Density Parity-Check code). The topology of the cluster including the physical arrangement of the nodes and disks within the cluster and status information for the nodes and disks within the cluster (e.g., information regarding disk fullness, disk performance, and disk age) may be used to identify the set of disks in which to store the plurality of data chunks.
1. A method for operating a data management system, comprising:
acquiring a first snapshot of a real or virtual machine;
generating a first plurality of data sets using the first snapshot of the real or virtual machine;
acquiring disk ages for a plurality of disks at a first point in time;
determining a plurality of estimated times to failure for the plurality of disks at the first point in time using the disk ages for the plurality of disks at the first point in time;
identifying a first subset of the plurality of disks less than all of the disks of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the first point in time;
identifying a second subset of the plurality of disks different from the first subset of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the first point in time;
storing a first data set of the first plurality of data sets using the first subset of the plurality of disks;
storing a second data set of the first plurality of data sets using the second subset of the plurality of disks;
acquiring a second snapshot of the real or virtual machine subsequent to acquiring the first snapshot of the real or virtual machine;
generating a second plurality of data sets using the second snapshot of the real or virtual machine;
acquiring disk ages for the plurality of disks at a second point in time subsequent to the first point in time;
determining a plurality of estimated times to failure for the plurality of disks at the second point in time using the disk ages for the plurality of disks at the second point in time;
identifying a third subset of the plurality of disks less than all of the disks of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the second point in time;
identifying a fourth subset of the plurality of disks different from the third subset of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the second point in time;
storing a third data set of the second plurality of data sets using the third subset of the plurality of disks; and
storing a fourth data set of the second plurality of data sets using the fourth subset of the plurality of disks.
2. The method of claim 1 , wherein:
the first subset of the plurality of disks is different from the third subset of the plurality of disks.
3. The method of claim 1 , wherein:
the identifying the first subset of the plurality of disks includes identifying a first grouping of the plurality of disks with an estimated time to failure associated with a first time period; and
the identifying the second subset of the plurality of disks includes identifying a second grouping of the plurality of disks with an estimated time to failure associated with a second time period different from the first time period.
4. The method of claim 3 , wherein:
the first grouping of the plurality of disks has an estimated time to failure between one year and two years and the second grouping of the plurality of disks has an estimated time to failure of less than one year.
5. The method of claim 3 , wherein:
the first time period does not overlap with the second time period.
6. The method of claim 1 , further comprising:
concurrently writing a third data chunk of the third data set to a third disk of the plurality of disks and writing a fourth data chunk of the fourth data set to a fourth disk of the plurality of disks.
7. The method of claim 6 , wherein:
the third disk comprises a third hard-disk drive; and
the fourth disk comprises a fourth hard-disk drive.
8. The method of claim 1 , wherein:
the generating the first plurality of data sets using the first snapshot of the real or virtual machine includes generating the first plurality of data sets using one of replication of data corresponding with the first snapshot or application of erasure coding techniques to the data corresponding with the first snapshot; and
the generating the second plurality of data sets using the second snapshot of the real or virtual machine includes generating the second plurality of data sets using one of replication of data corresponding with the second snapshot or application of erasure coding techniques to the data corresponding with the second snapshot.
9. The method of claim 1 , wherein:
the determining the plurality of estimated times to failure for the plurality of disks at the first point in time includes determining a plurality of mean-time-to-failure values for the plurality of disks using the disk ages for the plurality of disks at the first point in time.
10. The method of claim 1 , wherein:
the first snapshot comprises a full image snapshot of the virtual machine.
11. The method of claim 1 , further comprising:
determining a plurality of disk fullness values for the plurality of disks at the first point in time, the identifying the first subset of the plurality of disks includes identifying the first subset of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the first point in time and the plurality of disk fullness values for the plurality of disks at the first point in time.
12. A data management system, comprising:
a memory configured to store a first snapshot of a virtual machine and store a second snapshot of the virtual machine different from the first snapshot of the virtual machine; and
one or more processors configured to generate a first plurality of data sets using the first snapshot of the virtual machine and acquire disk ages for a plurality of disks at a first point in time, the one or more processors configured to determine a plurality of estimated times to failure for the plurality of disks at the first point in time based on the disk ages for the plurality of disks at the first point in time and identify a first subset of the plurality of disks less than all of the disks of the plurality of disks based on the plurality of estimated times to failure for the plurality of disks at the first point in time, the one or more processors configured to identify a second subset of the plurality of disks different from the first subset of the plurality of disks based on the plurality of estimated times to failure for the plurality of disks at the first point in time, the one or more processors configured write a first data set of the first plurality of data sets to the first subset of the plurality of disks and write a second data set of the first plurality of data sets to the second subset of the plurality of disks, the one or more processors configured to generate a second plurality of data sets using the second snapshot of the virtual machine and acquire disk ages for a plurality of disks at a second point in time subsequent to the first point in time, the one or more processors configured to determine a plurality of estimated times to failure for the plurality of disks at the second point in time based on the disk ages for the plurality of disks at the second point in time and identify a third subset of the plurality of disks less than all of the disks of the plurality of disks based on the plurality of estimated times to failure for the plurality of disks at the second point in time, the one or more processors configured to identify a fourth subset of the plurality of disks different from the third subset of the plurality of disks based on the plurality of estimated times to failure for the plurality of disks at the second point in time, the one or more processors configured to write a third data set of the second plurality of data sets to the third subset of the plurality of disks and write a fourth data set of the second plurality of data sets to the fourth subset of the plurality of disks.
13. The data management system of claim 12 , wherein:
the first subset of the plurality of disks is different from the third subset of the plurality of disks.
14. The data management system of claim 12 , wherein:
the one or more processors configured to identify a first grouping of the plurality of disks with an estimated time to failure associated with a first time period and identify a second grouping of the plurality of disks with an estimated time to failure associated with a second time period different from the first time period, the first subset of the plurality of disks comprises the first grouping of the plurality of disks, the second subset of the plurality of disks comprises the second grouping of the plurality of disks.
15. The data management system of claim 14 , wherein:
the first grouping of the plurality of disks has an estimated time to failure between one year and two years and the second grouping of the plurality of disks has an estimated time to failure of less than one year.
16. The data management system of claim 14 , wherein:
the first time period does not overlap with the second time period.
17. The data management system of claim 12 , wherein:
the one or more processors configured to concurrently write a third data chunk of the third data set to a third disk of the plurality of disks while a fourth data chunk of the fourth data set is written to a fourth disk of the plurality of disks.
18. The data management system of claim 17 , wherein:
the third disk comprises a third hard-disk drive; and
the fourth disk comprises a fourth hard-disk drive.
19. The data management system of claim 12 , wherein:
the one or more processors configured to determine a plurality of mean-time-to-failure values for the plurality of disks at the first point in time using the disk ages for the plurality of disks at the first point in time.
20. One or more storage devices containing processor readable code for programming one or more processors to perform a method for operating a data management system, the processor readable code comprising:
processor readable code configured to acquire a first snapshot of a real or virtual machine;
processor readable code configured to generate a first plurality of data sets using the first snapshot of the real or virtual machine;
processor readable code configured to acquire disk ages for a plurality of disks at a first point in time;
processor readable code configured to determine a plurality of estimated times to failure for the plurality of disks at the first point in time using the disk ages for the plurality of disks at the first point in time;
processor readable code configured to identify a first subset of the plurality of disks less than all of the disks of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the first point in time, the first subset of the plurality of disks comprises a first grouping of the plurality of disks with an estimated time to failure associated with a first time period;
processor readable code configured to identify a second subset of the plurality of disks different from the first subset of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the first point in time, the second subset of the plurality of disks comprises a second grouping of the plurality of disks with an estimated time to failure associated with a second time period different from the first time period;
processor readable code configured to store a first data set of the first plurality of data sets using the first subset of the plurality of disks;
processor readable code configured to store a second data set of the first plurality of data sets using the second subset of the plurality of disks;
processor readable code configured to acquire a second snapshot of the real or virtual machine subsequent to acquiring the first snapshot of the real or virtual machine;
processor readable code configured to generate a second plurality of data sets using the second snapshot of the real or virtual machine;
processor readable code configured to acquire disk ages for a plurality of disks at a second point in time subsequent to the first point in time;
processor readable code configured to determine a plurality of estimated times to failure for the plurality of disks at the second point in time using the disk ages for the plurality of disks at the second point in time;
processor readable code configured to identify a third subset of the plurality of disks less than all of the disks of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the second point in time;
processor readable code configured to identify a fourth subset of the plurality of disks different from the third subset of the plurality of disks using the plurality of estimated times to failure for the plurality of disks at the second point in time;
processor readable code configured to store a third data set of the second plurality of data sets using the third subset of the plurality of disks; and
processor readable code configured to store a fourth data set of the second plurality of data sets using the fourth subset of the plurality of disks, the first subset of the plurality of disks is different from the third subset of the plurality of disks.