Object store mirroring and garbage collection during synchronization of the object store
Techniques are provided for object store mirroring. Data within a storage tier of a node may be determined as being data to tier out to a primary object store based upon a property of the data. A first object is generated to comprise the data. A second object is generated to comprise the data. The first object is transmitted to the primary data store for storage in parallel with the second object being transmitted to a mirror object store for storage. Tiering of the data is designated as successful once acknowledgements are received from both the primary object that the first object was stored and the mirror object store that the second object was stored.
1. A method comprising:
attaching a first storage bucket of a primary object store to a node, wherein the node transmits data of a storage tier of the node to the primary object store for storage as objects within the first storage bucket;
attaching a second storage bucket of a mirror object store to the node, wherein the node transmits the data of the storage tier to the mirror object store for storage as objects within the second storage bucket;
and utilizing the objects within the first storage bucket to resynchronize the second storage bucket of the mirror object store,
wherein the second storage bucket comprises the objects mirroring the data of the objects within the first storage bucket,
wherein an object within the first storage bucket is skipped by the resynchronization based upon the object having a creating state indicating that the object has not yet been verified as successfully storing valid data within the primary object store,
and wherein during the resynchronization, a second object is deleted from the primary object store and the mirror object store by a garbage collection process.
2. The method of claim 1 , comprising:
storing a first object into the primary object store in parallel with storing a third object, as a replica of the first object, into the mirror object store.
3. The method of claim 1 , comprising:
attaching a third storage bucket of the primary object store to the node based upon a determination that a parameter of the first storage bucket is incorrect,
wherein a resync operation is performed to read a set of objects of the first storage bucket to create a second set of objects that are stored within the third storage bucket, and
wherein data of the second set of objects mirrors data of the set of objects.
4. The method of claim 1 , comprising:
transmitting the data from the node to the primary object store and the mirror object store in parallel.
5. The method of claim 4 , comprising:
attaching a third storage bucket of the primary object store to the node based upon a determination that performance of the first storage bucket is below a threshold, wherein a resync operation is performed to read a subset of a set of objects of the first storage bucket to create a second set of objects that are stored within the third storage bucket.
6. The method of claim 5 , comprising:
deleting the subset of the set of objects from the first storage bucket upon the resync operation completing.
7. The method of claim 1 , wherein the first storage bucket comprises a set of objects stored by the node using the data of the storage tier, and wherein the second storage bucket is initially created as a new empty storage bucket.
8. The method of claim 7 , comprising:
performing a resync operation to read the set of objects of the first storage bucket from the primary object store to create a second set of objects to store within the second storage bucket of the mirror object store.
9. The method of claim 8 , comprising:
skipping objects within the set of objects having a zero reference count.
10. The method of claim 8 , comprising:
generating a checkpoint to indicate a last object of the set of objects used to create and store an object within the second set of objects to the mirror object store,
wherein the resync operation uses the checkpoint to restart from the last object based upon the resync operation restarting.
11. The method of claim 8 , wherein the resync operation retains the data within the set of objects within an encrypted state when creating the second set of objects.
12. The method of claim 8 , comprising:
detaching and deleting the first storage bucket based upon the resync operation completing in order to complete a migration from the primary object store to the mirror object store.
13. The method of claim 8 , comprising:
detaching the first storage bucket based upon the resync operation completing.
14. The method of claim 8 , comprising:
deleting the first storage bucket upon the resync operation completing.
15. A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:
attach a first storage bucket of a primary object store to a node, wherein the node transmits data of a storage tier of the node to the primary object store for storage as objects within the first storage bucket;
attach a second storage bucket of a mirror object store to the node, wherein the node transmits the data of the storage tier to the mirror object store for storage as objects within the second storage bucket;
and utilize the objects within the first storage bucket to resynchronize the second storage bucket of the mirror object store, wherein the second storage bucket comprises the objects mirroring the data of the objects within the first storage bucket,
and wherein during the resynchronization, a second object is deleted from the primary object store and the mirror object store in parallel by a garbage collection process.
16. The non-transitory machine readable medium of claim 15 , wherein the first storage bucket comprises a set of objects stored by the node using the data of the storage tier, and wherein the second storage bucket is initially created as a new empty storage bucket.
17. The non-transitory machine readable medium of claim 16 , wherein the instructions cause the machine to:
perform a resync operation to read the set of objects of the first storage bucket from the primary object store to create a second set of objects to store within the second storage bucket of the mirror object store, wherein data of the second set of objects mirrors the data of the set of objects.
18. A computing device comprising:
a memory comprising machine executable code for performing a method;
and a processor coupled to the memory, the processor configured to execute the machine executable code to cause the computing device to:
attach a first storage bucket of a primary object store to a node, wherein the node transmits data of a storage tier of the node to the primary object store for storage as objects within the first storage bucket;
attach a second storage bucket of a mirror object store to the node, wherein the node transmits the data of the storage tier to the mirror object store for storage as objects within the second storage bucket;
and utilize the objects within the first storage bucket to resynchronize the second storage bucket of the mirror object store,
wherein the second storage bucket comprises the objects mirroring the data of the objects within the first storage bucket,
wherein an object within the first storage bucket is skipped by the resynchronization,
and wherein during the resynchronization, a second object is deleted from the primary object store and the mirror object store by a garbage collection process.
19. The computing device of claim 18 , wherein the first storage bucket comprises a set of objects stored by the node using the data of the storage tier, and wherein the second storage bucket is initially created as a new empty storage bucket, and wherein the machine executable code causes the processor to:
perform a resync operation to read the set of objects of the first storage bucket from the primary object store to create a second set of objects to store within the second storage bucket of the mirror object store, wherein data of the second set of objects mirrors the data of the set of objects.