Method for using catch-up logging to time-synchronize object stores during maintenance or recovery operations
Provided is a method of synchronizing a distributed object store including a plurality of data stores, the method including bringing a first data store of the data stores online after having been offline, choosing a second data store of the data stores, the second data store being online while the first data store was offline, reading, by the first data store, an entry from a catch-up log, and performing a data modification operation corresponding to the entry on the first data store to synchronize the plurality of data stores.
1. A method of synchronizing a distributed object store comprising a plurality of data stores, the method comprising:
performing a first operation at a first data store and at a second data store, such that the first data store and the second data store have a first state of an object;
performing a second operation on the object at the second data store, such that the second data store has a second state that is different from the first state;
generating an entry in a loci based on the second operation, the entry indicating a key associated with the object, and indicating the second state of the second data store as a result of the second operation;
performing a third operation on the object at the second data store, such that the second data store has the first state;
bringing the first data store having the first state online;
reading the entry indicating the second state of the second data store; and
based at least in part on:
accessing the second data store, which is based on the reading of the entry indicating the second state; and
determining that the second data store has the first state,
maintaining the first state of the first data store.
2. The method of claim 1 , further comprising causing the first data store to go offline in accordance with a maintenance operation.
3. The method of claim 1 , further comprising storing the log on multiple nodes of a cluster underlying the distributed object store.
4. The method of claim 1 , wherein the log comprises an unordered set of entries corresponding to respective data modification operations.
5. The method of claim 1 , wherein the entry indicates a type of data modification operation.
6. The method of claim 1 , wherein:
the entry comprises a delete entry;
the second operation comprises a delete operation; and
the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is contained in the second data store.
7. The method of claim 1 , wherein:
the entry comprises a write entry;
the second operation comprises a write operation; and
the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is absent from the second data store.
8. A distributed object store, as a non-transitory storage medium, comprising a plurality of data stores, the distributed object store being configured to be synchronized by:
performing a first operation at a first data store and at a second data store, such that the first data store and the second data store have a first state of an object;
performing a second operation on the object at the second data store, such that the second data store has a second state that is different from the first state;
generating an entry in a loci based on the second operation, the entry indicating a key associated with the object, and indicating the second state of the second data store as a result of the second operation;
performing a third operation on the object at the second data store, such that the second data store has the first state;
bringing the first data store having the first state online;
reading the entry indicating the second state of the second data store; and
based at least in part on:
accessing the second data store, which is based on the reading of the entry indicating the second state; and
determining that the second data store has the first state,
maintaining the first state of the first data store.
9. The distributed object store of claim 8 , wherein the distributed object store is further configured to be synchronized by causing the first data store to go offline in accordance with a maintenance operation.
10. The distributed object store of claim 8 , wherein the distributed object store is further configured to be synchronized by storing the log on multiple nodes of a cluster underlying the distributed object store.
11. The distributed object store of claim 8 , wherein the log comprises an unordered set of entries corresponding to respective data modification operations.
12. The distributed object store of claim 8 , wherein the entry indicates a type of data modification operation.
13. The distributed object store of claim 8 , wherein:
the entry comprises a delete entry;
the second operation comprises a delete operation; and
the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is contained in the second data store.
14. The distributed object store of claim 8 , wherein:
the entry comprises a write entry
the second operation comprises a write operation; and
the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is absent from the second data store.
15. A distributed object store system comprising a non-transitory computer readable medium having computer code that, when executed on a processor, implements a method of synchronizing a distributed object store comprising a plurality of data stores, the method comprising:
performing a first operation at a first data store and at a second data store, such that the first data store and the second data store have a first state of an object;
performing a second operation on the object at the second data store, such that the second data store has a second state that is different from the first state;
generating an entry in a loci based on the second operation, the entry indicating a key associated with the object, and indicating the second state of the second data store as a result of the second operation;
performing a third operation on the object at the second data store, such that the second data store has the first state;
bringing the first data store having the first state online;
reading the entry indicating the second state of the second data store; and
based at least in part on:
accessing the second data store, which is based on the reading of the entry indicating the second state; and
determining that the second data store has the first state,
maintaining the first state of the first data store.
16. The distributed object store system of claim 15 , wherein the computer code, when executed by the processor, further implements the method of synchronizing the distributed object store by causing the first data store to go offline in accordance with a maintenance operation.
17. The distributed object store system of claim 15 , wherein the computer code, when executed by the processor, further implements the method of synchronizing the distributed object store by storing the log on multiple nodes of a cluster underlying the distributed object store.
18. The distributed object store system of claim 15 , wherein the log comprises an unordered set of entries corresponding to respective data modification operations.
19. The distributed object store system of claim 15 , wherein the entry indicates a type of data modification operation.
20. The distributed object store system of claim 15 , wherein:
the entry comprises a delete entry
the second operation comprises a delete operation; and
the maintaining the first state of the first data store comprises checking, by the first data store, the second data store to determine that the object is contained in the second data store.