Recovery after data loss in a reliable distributed computing system
Real system devices (RSD's) emulate virtual system devices (VSD's); system elements can use RSD's and manipulate VSD's, while users can only use VSD's. Identifying when RSD's have erroneous data. Providing a mapping of RSD's to VSD's, to recover a maximum set of VSD's. Providing a mapping of VSD's to RSD's, to recover a specified most valuable VSD. Reconstructing VSD's from RSD's having erroneous data, and coordinating with a user's assistance to couple RSD's in an order allowing data recovery. Metadata is maintained coupling data identifiable on VSD's with corresponding data on RSD's; it is possible to trace connections between RSD's and VSD's even without complete data.
1. A method,
in a reliable distributed computing system, said system including one or more virtual machines, each said virtual machine having access to a plurality of virtual storage devices, each said virtual machine operating under the control of a guest operating system,
of recovering a largest amount of data after a loss of data in one or more virtual storage devices,
said method including steps of
maintaining metadata associating one or more said real storage devices with one or more said virtual storage devices;
after said loss of data, in response to said metadata, receiving an instruction from a user selecting one of: an instruction to recover a set of particular virtual storage devices, an instruction to recover a largest number of virtual storage devices;
when said instruction from the user includes said instruction to recover a set of particular virtual storage devices, determining a set of real storage devices from which to recover a largest number of file storage units associated with those particular virtual storage devices, and determining an order in which to couple said set of real storage devices to the reliable distributed computing system;
when said instruction from the user includes said instruction to recover a largest number of virtual storage devices, determining a set of real storage devices from which to recover a largest number of said virtual storage devices, and determining an order in which said set of real storage devices should be coupled to the reliable distributed computing system;
informing said user of which real storage device to next couple to the reliable distributed computing system;
reading still-good data from said real storage device that was next coupled;
when said instruction from the user includes said instruction to recover a set of particular virtual storage devices, reconstructing the particular virtual storage device;
when said instruction from the user includes said instruction to recover a largest number of virtual storage devices, reconstructing the largest number of virtual storage devices, as directed by the user.
2. A method as in claim 1 , including steps of
repeating said steps of determining a set of real storage devices, and determining an order to couple said set of real storage devices to said reliable distributed computing system.
3. A method as in claim 1 , including steps of
determining one or more disk slots associated with said reliable distributed computing system in which to couple said set of real storage devices.
4. A method as in claim 1 , wherein
each said virtual storage device has its data maintained with redundant data;
wherein
said loss of data still allows said reliable distributed computing system to recover data on each said virtual storage device in response to said redundant data.
5. A method as in claim 4 , wherein
said redundant data includes one or more of:
error-correcting codes for said data;
multiple copies of said data;
spread-spectrum encoding of said data.