System for live-migration and automated recovery of applications in a distributed system
View Patent ↗A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilization level.
1. Computer software which, when executed by appropriate processing means, causes the processing means to implement a method for replicating a filesystem between a first server and a second server prior to and following a partition between the first server and the second server, the method comprising:
at the first server, taking snapshots of a current state of the filesystem at predetermined points in time following modification of the filesystem, each snapshot recording differences between the current state of the filesystem on the server and the state of the filesystem on the server at the time point of a previous snapshot;
continually replicating the snapshots taken on the first server to the second server as soon as they are taken;
upon detection of a partition, both the first and the second server becoming masters for the filesystem and accepting new modifications to the filesystems;
after recovery of the partition, performing an update process to update the filesystem, the update process comprising:
identifying which of the first server and the second server contains the most current version of the filesystem;
nominating the server identified as containing the most current version of the filesystem as the master server and the other server as the slave server;
identifying a snapshot that is common to both the master server and the slave server; and
replicating subsequent snapshots from the master server to the slave server.
2. Computer software according to claim 1 wherein identifying which of the first server and the second server contains the most current version of the filesystem comprises calculating a centre of mass metric for the version of the filesystem on each of the servers, the centre of mass metric representing the average age of the snapshots of the filesystem on each server and the number of changes to the filesystem represented by the snapshots on each server.
3. Computer software according to any preceding claim 1 wherein identifying which of the first server and the second server contains the most current version of the filesystem further comprises identifying a set of snapshots of the filesystem that for each server, each set of snapshots containing snapshots only present on that server, and calculating the centre of mass metric for each server based on that server's set of snapshots.
4. A system for replicating a filesystem between a first server and a second server prior to and following a partition between the first server and the second server, the system comprising:
snapshotting means for taking snapshots of a current state of the filesystem on the first server at predetermined points in time following modification of the filesystem, each snapshot recording differences between the current state of the filesystem on the server and the state of the filesystem on the server at the time point of a previous snapshot;
replicator means for continually replicating the snapshots taken on the first server to the second server as soon as they are taken;
detection means configured such that upon detection of a partition, both the first and the second server become masters for the filesystem and accept new modifications to the filesystems;
updating means configured to perform an update process to update the filesystem after recovery of the partition, the update process comprising:
identifying which of the first server and the second server contains the most current (i.e. most valuable) version of the filesystem;
nominating the server identified as containing the most current version of the filesystem as the master server and the other server as the slave server;
identifying a snapshot that is common to both the master server and the slave server; and
replicating subsequent snapshots from the master server to the slave server.
5. A system according to claim 4 wherein identifying which of the first server and the second server contains the most current (i.e. most valuable) version of the filesystem comprises calculating a centre of mass metric for the version of the filesystem on each of the servers, the centre of mass metric representing the average age of the snapshots of the filesystem on each server and the number of changes to the filesystem represented by the snapshots on each server.
6. A system according to claim 4 wherein identifying which of the first server and the second server contains the most current (i.e. most valuable) version of the filesystem further comprises identifying a set of snapshots of the filesystem that for each server, each set of snapshots containing snapshots only present on that server, and calculating the centre of mass metric for each server based on that server's set of snapshots.
7. A system according to claim 4 wherein the update process further comprises storing the snapshots of the slave server that were taken after the common snapshot.
8. A system according to claim 4 further comprising storage means for storing the snapshots taken of the filesystem such that a previous snapshot of the filesystem can be selected by a user from the stored snapshots to restore the system to its state at the time of the selected snapshot.
9. A system according to claim 4 wherein the previous snapshot of the filesystem can be selected by means of a user interface presented to the user.