Object and sequence number management
Techniques are provided for orphan object detection, invalid sequence number detection, and asynchronous object cleanup. A storage system may store data within one or more tiers of storage, such as a storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Orphan objects, within the remote object store, that are no longer used by the storage system may be detected and/or deleted. When an aggregate of volumes is deleted, corresponding objects, within the remote object store, may be identified and/or deleted. Invalid sequence numbers (e.g., lost or corrupt sequence numbers locally maintained in a metafile) assigned to objects within the remote object store may be identified, deleted, and/or fixed.
1. A method comprising:
deleting data, associated with a composite aggregate, from a storage tier based upon a delete request received from a requestor to delete the composite aggregate, wherein the composite aggregate is composed of the data in the storage tier and objects stored within a remote object store and tracked using a metafile;
in response to deleting the data from the storage tier, transmitting an acknowledgment back to the requestor that the delete request was successfully performed;
subsequent to deleting the data and transmitting the acknowledgement back to the requestor, performing an asynchronous object cleanup operation, comprising:
sending a request to the remote object store for a list of object names of objects having object names associated with a volume identifier of a volume of the composite aggregate; and
issuing delete commands to the remote object store to delete the objects having the object names within the list of object names received from the remote object store; and
performing a metafile validation for the metafile, comprising:
comparing sequence numbers assigned to objects created within the remote object store for the volume with sequence numbers within the metafile to identify missing sequence numbers not specified in the metafile; and
adding the missing sequence numbers as valid sequence numbers to the metafile to repair the metafile.
2. The method of claim 1 , comprising assigning an object name to an object, wherein the object name comprises a prefix derived from the volume identifier of the volume, of the composite aggregate, to which the object belongs.
3. The method of claim 1 , comprising:
assigning the sequence numbers as monotonically increasing numbers to the objects created within the remote object store for the volume.
4. The method of claim 1 ,
assigning an object name to an object, wherein the object name is derived from a hash of the volume identifier and a sequence number assigned to the object;
mapping, within the metafile, the object name to an object identifier; and
utilizing the mapping in the metafile to identify and access the object within the remote object store.
5. The method of claim 1 , comprising:
storing a list of volume identifiers within a cluster-wide database accessible to a plurality of nodes of a storage environment that hosts the storage tier but does not host the remote object tier that is hosted by a third party provider.
6. The method of claim 5 , comprising:
deleting the list of volume identifiers from the cluster-wide database based upon objects associated with a set of volumes being deleted from the remote object store.
7. A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:
delete data, associated with a composite aggregate, from a storage tier based upon a delete request received from a requestor to delete the composite aggregate, wherein the composite aggregate is composed of the data in the storage tier and objects stored within a remote object store; and
in response to acknowledging successful completion of the delete request before the objects of the composite aggregate are deleted from the remote object store, perform an asynchronous object cleanup operation, comprising:
sending a request to the remote object store for a list of object names of objects having object names associated with a volume identifier of a volume of the composite aggregate;
issuing delete commands to the remote object store to delete the objects having the object names within the list of object names received from the remote object store;
making storage space consumed by the composite aggregate available for use by the requestor before the objects of the composite aggregate are deleted from the remote object store by the asynchronous object cleanup operation.
8. The non-transitory machine readable medium of claim 7 , wherein an object name of an object comprises a prefix derived from the volume identifier of the volume, of the composite aggregate, to which the object belongs.
9. The non-transitory machine readable medium of claim 7 , comprising:
assigning an object name to an object, wherein the object name is derived from a hash of the volume identifier and a sequence number assigned to the object;
mapping, within a metafile, the object name to an object identifier; and
utilizing the mapping in the metafile to identify and access the object within the remote object store.
10. The non-transitory machine readable medium of claim 7 , wherein the instructions cause the machine to:
maintain a metafile used to locate and access objects in the remote object store, wherein the metafile is populated with sequence numbers assigned to the objects.
11. The non-transitory machine readable medium of claim 7 , wherein an object name of an object is derived from a hash of the volume identifier and a sequence number assigned to the object.
12. The non-transitory machine readable medium of claim 7 , wherein the instructions cause the machine to:
store a list of volume identifiers within a cluster-wide database accessible to a plurality of nodes of a storage environment that hosts the storage tier but does not host the remote object tier that is hosted by a third party provider.
13. The non-transitory machine readable medium of claim 12 , wherein the instructions cause the machine to:
delete the list of volume identifiers from the cluster-wide database based upon objects associated with a set of volumes being deleted from the remote object store.
14. A computing device comprising:
a memory comprising machine executable code; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
perform an asynchronous object cleanup operation in response to data of a composite aggregate being deleted from a storage tier based upon a delete request received from a requestor to delete the composite aggregate, wherein the composite aggregate is composed of the data in the storage tier and objects stored within a remote object store, wherein the asynchronous object cleanup operation is performed after acknowledging successful deletion of the composite aggregate to the requestor before the objects are deleted from the remote object store, and wherein the asynchronous object cleanup operation comprises:
sending a request to the remote object store for a list of object names of the objects of the composite aggregate; and
issuing delete commands to the remote object store to delete the objects having the object names within the list of object names received from the remote object store; and
perform a metafile validation for a metafile, comprising:
comparing sequence numbers assigned as monotonically increasing numbers to objects created within the remote object store for the composite aggregate with sequence numbers within a metafile to identify invalid sequence numbers not specified in the metafile; and
adding the invalid sequence numbers as valid sequence numbers to the metafile to repair the metafile.
15. The computing device of claim 14 , wherein an object name of an object comprises a prefix derived from a volume identifier of a volume, of the composite aggregate, to which the object belongs, wherein the object name is within the list of object names.
16. The computing device of claim 14 , wherein an object name of an object is derived from a volume identifier and a sequence number assigned to the object.
17. The computing device of claim 16 , wherein the machine executable code cause the processor to:
maintain the metafile used to locate and access objects in the remote object store, wherein the metafile is populated with mappings between object IDs and object names of the objects.
18. The computing device of claim 14 , wherein an object name of an object is derived from a hash of a volume identifier and a sequence number assigned to the object.
19. The computing device of claim 14 , wherein the machine executable code cause the processor to:
store the list of volume identifiers within a cluster-wide database accessible to a plurality of nodes of a storage environment that hosts the storage tier but does not host the remote object tier that is hosted by a third party provider.
20. A method, comprising:
deleting data, associated with a composite aggregate, from a storage tier based upon a delete request received from a requestor to delete the composite aggregate, wherein the composite aggregate is composed of the data in the storage tier and objects stored within a remote object store; and
in response to acknowledging successful completion of the delete request before the objects of the composite aggregate are deleted from the remote object store, performing an asynchronous object cleanup operation, comprising:
sending a request to the remote object store for a list of object names of objects having object names associated with a volume identifier of a volume of the composite aggregate;
issuing delete commands to the remote object store to delete the objects having the object names within the list of object names received from the remote object store; and
making storage space consumed by the composite aggregate available for use by the requestor before the objects of the composite aggregate are deleted from the remote object store by the asynchronous object cleanup operation.
21. The method of claim 20 , wherein an object name of an object comprises a prefix derived from the volume identifier of the volume, of the composite aggregate, to which the object belongs.
22. The method of claim 20 , comprising:
assigning an object name to an object, wherein the object name is derived from a hash of the volume identifier and a sequence number assigned to the object;
mapping, within a metafile, the object name to an object identifier; and
utilizing the mapping in the metafile to identify and access the object within the remote object store.
23. The method of claim 20 , comprising:
maintaining a metafile used to locate and access objects in the remote object store, wherein the metafile is populated with sequence numbers assigned to the objects.
24. The method of claim 20 , wherein an object name of an object is derived from a hash of the volume identifier and a sequence number assigned to the object.
25. The method of claim 20 , comprising:
storing a list of volume identifiers within a cluster-wide database accessible to a plurality of nodes of a storage environment that hosts the storage tier but does not host the remote object tier that is hosted by a third party provider.
26. The method of claim 25 , comprising:
deleting the list of volume identifiers from the cluster-wide database based upon objects associated with a set of volumes being deleted from the remote object store.