VIRTUAL MULTI-CLUSTER CLOUDS
An improved scalable object storage system allows multiple clusters to work together. Users working with a first cluster, or with a multi-cluster gateway, can ask for services and have the request or data transparently proxied to a second cluster. This gives transparent cross-cluster replication, as well as multi-cluster compute or storage farms based upon spot availability or various provisioning policies. Vendors providing a cloud storage “frontend” can provide multiple backends simultaneously. In one embodiment, a multi-cluster gateway can have a two, three, or higher-level ring that transparently matches an incoming request with the correct cluster. In the ring, a request is first mapped to an abstract “partition” based on a consistent hash function, and then one or more constrained mappings map the partition number to an actual resource. In another embodiment, the multi-cluster gateway is a dumb gateway, and the rings are located only at the cluster level.
1 . (canceled)
2 . A method of specifying container synchronization targets and authorization using a plurality of attributes, the method comprising:
setting a first attribute of a first container to a first uniform resource locator (URL) of a second container, the first container storing a first plurality of objects, and the second container storing a second plurality of objects;
setting a second attribute of the first container to a key value;
setting a first attribute of the second container to a second URL of the first container;
setting a second attribute of the second container to the key value, wherein setting the first and second attributes of the first container and setting the first and second attributes of the second container set the first and second pluralities of objects to be synchronized, but does not yet synchronize the first and second pluralities of objects; and
replicating a first object of the first plurality of objects to the second container referenced by the first URL.
3 . The method of claim 2 , wherein the replicating includes replicating, by a background process, the first object to the second container.
4 . The method of claim 2 , further comprising:
receiving a request to replicate the first object to the second container, wherein the replicating includes replicating in response to receiving the request.
5 . The method of claim 2 , further comprising:
receiving a request to add the first object to the first container;
inserting the first object into the first container, wherein the replicating includes replicating in response to receiving the request.
6 . The method of claim 2 , further comprising:
scanning, by a first daemon running in the first container, one or more container databases for containers configured to synchronize; and
sending one or more synchronization updates to one or more entities, the one or more synchronization updates based on newly detected row identifiers in the scanned container databases.
7 . The method of claim 2 , wherein the first container has a higher priority attribute value than the second container, wherein a container having a higher priority attribute value than another container is allotted a greater amount of time to synchronize.
8 . The method of claim 2 , further comprising:
maintaining a first synchronization point in a local container database of the first container, the first synchronization point specifying a newest row identifier known; and
maintaining a second synchronization point in the local container database of the first container, the second synchronization point specifying the newest row identifier for which synchronization updates have been sent.
9 . The method of claim 8 , wherein the maintaining the first synchronization point includes maintaining, by a first daemon running in the first container, the first synchronization point and maintaining second synchronization point includes maintaining, by the first daemon running in the first container, the second synchronization point.
10 . The method of claim 8 , wherein an updated row that is newer than the first and synchronization points causes a synchronization update to be sent.
11 . The method of claim 2 , wherein the key is a user-provided shared secret that is keyed to a particular account.
12 . A system for specifying container synchronization targets and authorization using a plurality of attributes, the system comprising:
a first container server that sets a first attribute of a first container to a first uniform resource locator (URL) of a second container and sets a second attribute of the first container to a key value, wherein the first container stores a first plurality of objects, and the second container stores a second plurality of objects;
a second container server that sets a first attribute of a second container to a second uniform resource locator (URL) of the first container and sets a second attribute of the second container to the key value; and
a replicator that replicates a first object of the first plurality of objects to the second container referenced by the first URL.
13 . The system of claim 12 , wherein the replicator is a background process executing in the first container server.
14 . The system of claim 12 , wherein the replicator receives a request to replicate the first object to the second container, and in response to the request, the replicator replicates the first object to the second container.
15 . The system of claim 12 , wherein the replicator receives a request to add the first object to the first container, and in response to the request, the replicator replicates the first object to the second container.
16 . The system of claim 12 , wherein the replicator scans one or more container databases for containers configured to synchronize and sends one or more synchronization updates to one or more entities, and wherein the one or more synchronization updates is based on newly detected row identifiers in the scanned container databases.
17 . The system of claim 12 , wherein the first container has a higher priority attribute value than the second container, wherein a container having a higher priority attribute value than another container is allotted a greater amount of time to synchronize.
18 . The system of claim 12 , wherein the replicator maintains a first synchronization point in a local container database of the first container and maintains a second synchronization point in the local container database of the first container, and wherein the first synchronization point specifies a newest row identifier known and the second synchronization point specifies the newest row identifier for which synchronization updates have been sent.
19 . The system of claim 18 , wherein the replicator is a daemon running in the first container.
20 . The system of claim 18 , wherein an updated row that is newer than the first and synchronization points causes a synchronization update to be sent.