Virtual devices in a reliable distributed computing system
Providing differing degrees of reliability or security for distinct devices within a reliable distributed system. Allowing virtual machines to operate in a reliable distributed system without either exposing the actual available devices to guest operating systems, or requiring the guest OS to maintain, or to maintain information about, reliability or security. Methods are responsive to a hierarchy of possible reliability or security failures, to guard more carefully against simultaneous failures of reliability or breaches of security, without additional work or delay. Functions invoked by the user that relate to reliability and security are hidden, so the user can specify a degree of reliability or security without having to implement those requirements themselves. Providing a uniform set of resources available to all users, separating out those individual resources that are allocated to particular users, and emulating particular devices at the request of those particular users. Users do not have access to “real” devices the distributed system can use, only “virtual” devices presented by the distributed system, where those virtual devices have properties similar, but not necessarily equal, to real devices.
1. A method, in a fault-tolerant distributed computing system including one or more virtual machines, each particular virtual machine having one or more user applications with access to one or more virtual storage devices, of providing degrees of reliability associated with each virtual storage device in the fault-tolerant distributed computing system, each degree of reliability being selected by a user application for a portion of an associated virtual storage device, the method including steps of:
emulating the virtual storage devices using one or more real storage devices, the real storage devices being subject to potential failure, and the virtual storage devices being maintained at the defined degrees of reliability;
receiving and responding to messages from the user applications executing at virtual machines, the messages from a selected user application including setting metadata defining a degree of reliability associated with said portion of a virtual storage device for that virtual machine;
maintaining metadata indicating a first and second failure group, each failure group indicating at least a portion of a real storage device, the metadata including which cross-products of failure groups and virtual storage devices are assigned for use by real storage devices, each virtual storage device disposed to be maintained across more than one failure group; and
maintaining those portions of the virtual storage devices at associated portions of those real storage devices, wherein no two or more portions of each virtual storage device is assigned to the same one of first set of failure groups and to the same one of the second set of failure groups.
2. A method as in claim 1 , including steps of
providing access to the redundant information for a virtual storage device despite multiple concurrent failures of real storage devices in a selected failure group.
3. A method as in claim 1 , including steps of
maintaining data and redundant information responsive to that data, from which that data can be recovered in response to loss of copies of that data, for each portion of those virtual storage devices, wherein an amount of redundant information is associated with the degree of reliability;
wherein the data and redundant information are maintained at two or more separate ones of those failure groups.
4. A method as in claim 1 ,
wherein the selected failure groups define a hierarchy of failure groups, describing at least a first level of failure groups, and at least a second level of failure groups, the second level of failure groups including groups of the first level of failure groups.
5. A method as in claim 4 , wherein
the first level of failure groups includes real storage devices associated with the same node in the fault-tolerant distributed system, the fault-tolerant distributed system having multiple nodes coupled to a shared memory; and
the second level of failure groups includes multiple independent nodes coupled to a single point of failure, the single point of failure including one or more of: a joint communication link, a joint power supply.
6. A method as in claim 4 , wherein
the first level of failure groups includes nodes coupled to a single point of failure in the fault-tolerant distributed system; and
the second level of failure groups includes real storage devices coupled to multiple independent nodes in the same room.
7. A method as in claim 4 , wherein
the first level of failure groups includes nodes coupled to a single point of failure in the fault-tolerant distributed system; and
the second level of failure groups includes real storage devices coupled to multiple independent nodes in the same building.
8. A method as in claim 1 , including steps of
maintaining information coupling identified portions of those virtual storage devices to their associated portions of real storage devices; and
maintaining information coupling those portions of real storage devices to their associated failure groups.
9. A method as in claim 1 , wherein
the first and second sets of failure groups each being responsive to a distinct failure mode.
10. A method as in claim 1 , wherein
in response to degrees of reliability selected by user applications, each distinct virtual machine is associated with a different amount of redundant information.
11. A method as in claim 1 , wherein
in response to degrees of reliability selected by user applications, each distinct virtual machine is associated with a different type of error-correcting code to use with data for each data block.
12. A fault-tolerant distributed computing system including:
one or more virtual machines, each particular virtual machine having one or more user applications with access to one or more virtual storage devices, the system providing degrees of reliability associated with each virtual storage device in the fault-tolerant distributed computing system, each degree of reliability being selected by a user application for a portion of an associated virtual storage device;
one or more nodes including one or more processors; and
a non-transitory computer-readable medium having stored thereon instructions directing the one or more processors to:
emulate the virtual storage devices using one or more real storage devices, the real storage devices being subject to potential failure, and the virtual storage devices being maintained at the defined degrees of reliability;
receive and respond to messages from the user applications executing at virtual machines, the messages from a selected user application including setting metadata defining a degree of reliability associated with said portion of a virtual storage device for that virtual machine;
maintaining metadata indicating a first and second failure group, each failure group indicating at least a portion of a real storage device, the metadata including which cross-products of failure groups and virtual storage devices are assigned for use by real storage devices, each virtual storage device disposed to be maintained across more than one failure group; and
maintain those portions of the virtual storage devices to associated portions of those real storage devices, wherein no two or more portions of each virtual storage device is assigned to the same one of first set of failure groups and to the same one of the second set of failure groups.
13. A system as in claim 12 , including instructions
to provide access to the redundant information for a virtual storage device despite multiple concurrent failures of real storage devices in a selected failure group.
14. A system as in claim 12 , including instructions
to maintain data and redundant information responsive to that data, from which that data can be recovered in response to loss of copies of that data, for each portion of those virtual storage devices, wherein an amount of redundant information is associated with the degree of reliability;
wherein the data and redundant information are maintained at two or more separate ones of those failure groups.
15. A system as in claim 12 ,
wherein the selected failure groups define a hierarchy of failure groups, describing at least a first level of failure groups, and at least a second level of failure groups, the second level of failure groups including groups of the first level of failure groups.
16. A system as in claim 15 , wherein
the first level of failure groups includes real storage devices associated with the same node in the fault-tolerant distributed system, the fault-tolerant distributed system having multiple nodes coupled to a shared memory; and
the second level of failure groups includes multiple independent nodes coupled to a single point of failure, the single point of failure including one or more of: a joint communication link, a joint power supply.
17. A system as in claim 15 , wherein
the first level of failure groups includes nodes coupled to a single point of failure in the fault-tolerant distributed system; and
the second level of failure groups includes real storage devices coupled to multiple independent nodes in the same room.
18. A system as in claim 15 , wherein
the first level of failure groups includes nodes coupled to a single point of failure in the fault-tolerant distributed system; and
the second level of failure groups includes real storage devices coupled to multiple independent nodes in the same building.
19. A system as in claim 12 , including instructions
to maintain information coupling identified portions of those virtual storage devices to their associated portions of real storage devices; and
to maintain information coupling those portions of real storage devices to their associated failure groups.
20. A system as in claim 12 , wherein
the first and second sets of failure groups each being responsive to a distinct failure mode.
21. A system as in claim 12 , wherein
in response to degrees of reliability selected by user applications, each distinct virtual machine is associated with a different amount of redundant information.
22. A system as in claim 12 , wherein
in response to degrees of reliability selected by user applications, each distinct virtual machine is associated with a different type of error-correcting code to use with data for each data block.