System for rebuilding dispersed data
View Patent ↗A digital data file storage system is disclosed in which original data files to be stored are dispersed using some form of information dispersal algorithm into a number of file “slices” or subsets in such a manner that the data in each file share is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other file shares. These file shares are stored on separate digital data storage devices as a way of increasing privacy and security. As dispersed file shares are being transferred to or stored on a grid of distributed storage locations, various grid resources may become non-operational or may operate below at a less than optimal level. When dispersed file shares are being written to a dispersed storage grid which not available, the grid clients designates the dispersed data shares that could not be written at that time on a Rebuild List. In addition when grid resources already storing dispersed data become non-available, a process within the dispersed storage grid designates the dispersed data shares that need to be recreated on the Rebuild List. At other points in time a separate process reads the set of Rebuild Lists used to create the corresponding dispersed data and stores that data on available grid resources.
1. A method comprising the steps of:
(a) assembling a list of unavailable storage nodes;
(b) compiling a list of affected data segments contained on the unavailable storage nodes;
(c) determining a list of data slices needed to rebuild the affected data segments;
(d) reading the data slices needed to rebuild the affected data segments from associated storage nodes;
(e) rebuilding the affected data segments from the data slices;
(f) creating new data slices from the rebuilt data segment using an information dispersal algorithm; and
(g) writing the new data slices associated with the unavailable storage nodes to different storage nodes;
wherein the step of creating new data slices comprises partitioning a data segment into a set of data slices, and creating an encoded data subset by summing a modulo of at least two data slices within a collection of data slices; and wherein the step of writing further includes storing each data slice along with its corresponding encoded data subset in the storage node; and
wherein the step of rebuilding the data segments comprises combining a subset of the data slices and the encoded data subsets required to reproduce the data segment from less than all of the original data slices.
2. The method of claim 1 wherein the step of writing the data slices into storage nodes comprises writing one of the data slices and its associated encoded data subset into a storage node that is located in a separate facility from at least some of the other associated storage nodes.
3. A method of rebuilding data stored on a dispersed data storage network comprising a plurality of networked computers including a plurality of storage nodes, each of said storage nodes storing a plurality of data slices, whereby n of said data slices are associated with a corresponding file, and whereby m of said associated data slices are required to reconstruct said corresponding file, and further whereby m is less than n, said method operating on a computer and comprising the steps of:
(a) assembling a list of unusable storage nodes;
(b) compiling a list of affected files based on the list of unusable storage nodes;
(c) for each listed file:
(i) assembling a list of m data slices needed to rebuild the affected file wherein each listed data slice is stored on a separate available storage node;
(ii) reading the listed data slices from the corresponding storage nodes;
(iii) assembling the listed file from the data slices using an information dispersal algorithm;
(iv) creating n new data slices from the rebuilt file using an information dispersal algorithm; and
(v) writing the n new data slices associated with the unavailable storage nodes to different available storage nodes.
4. The method of claim 3 wherein each data slice comprises a data subset and a coded data subset.
5. A dispersed data storage network comprising a plurality of networked computers including a plurality of slice servers, each of said slice servers storing a plurality of data slices, whereby n of said data slices are associated with a corresponding file, and whereby m of said associated data slices are required to reconstruct said corresponding file, and further whereby m is less than n, said dispersed data storage network further comprising:
(a) a computer coupled to said plurality of networked computers, said computer running a rebuilder application for:
(i) assembling a list of unusable storage nodes;
(ii) compiling a list of affected files based on the list of unusable storage nodes; and
(iii) for each listed file:
(1) assembling a list of m data slices needed to rebuild the affected file wherein each listed data slice is stored on a separate available storage node;
(2) reading the listed data slices from the corresponding storage nodes;
(3) assembling the listed file from the read data slices using an information dispersal algorithm;
(4) creating n new data slices from the assembled file using an information dispersal algorithm; and
(5) writing the n new data slices associated with the unavailable storage nodes to different available storage nodes.
6. The dispersed data storage network of claim 5 wherein each data slice comprises a data subset and a coded data subset.