Object and sequence number management
Techniques are provided for orphan object detection, invalid sequence number detection, and asynchronous object cleanup. A storage system may store data within one or more tiers of storage, such as a storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Orphan objects, within the remote object store, that are no longer used by the storage system may be detected and/or deleted. When an aggregate of volumes is deleted, corresponding objects, within the remote object store, may be identified and/or deleted. Invalid sequence numbers (e.g., lost or corrupt sequence numbers locally maintained in a metafile) assigned to objects within the remote object store may be identified, deleted, and/or fixed.
1 . A method, comprising:
creating a composite aggregate composed of a volume exposed to clients, wherein data of the volume is stored across a storage tier and within objects of a remote object store;
creating an object comprising an object header and an object page for storing to a data chunk, of the volume, being moved from the storage tier to the remote object store, wherein the creating comprises:
generating an object name for the object based upon a volume identifier of the volume referencing the data chunk;
populating the object header with an identifier of the object name;
populating the object page with the data chunk and a context associated with the object page; and
storing the object into the remote object store;
comparing the object name, determined from the identifier within the object header, with the volume identifier of the volume to determine whether the object name and volume identifier match;
in response to the object name and volume identifier matching, providing access to the data chunk according to the context associated with the object page; and
in response to the object name and volume identifier not matching, determining that the object is no longer referenced by the volume.
2 . The method of claim 1 , further comprising:
assigning the object name based upon the volume identifier and a sequence number assigned to the object for referencing the object; and
utilizing the sequence number within the object name to validate a metafile used by a storage system to track objects, within the remote object store, storing data of the volume.
3 . The method of claim 2 , further comprising:
creating a hash for the object storing the data chunk, wherein the hash is created from both the volume identifier of the volume referencing the data chunk and the sequence number assigned to the object for referencing the object; and
deriving the object name for the object based upon the hash, wherein the object name is verified by reading the hash from the object within the remote object store.
4 . The method of claim 1 , further comprising:
generating the identifier of the object name based upon a hash of a buff tree universal identifier used as the volume identifier; and
in response to storing the object into the remote object store, reading the hash back from the object within the remote object store to verify the object.
5 . The method of claim 1 , further comprising:
in response to the object name and the volume identifier matching, specifying that the object is a valid object used by the volume;
comparing a sequence number of the valid object with a list of sequence numbers within a metafile; and
in response to the sequence number not matching at least one sequence number within the list of sequence numbers, designating the metafile as corrupt.
6 . The method of claim 1 , further comprising:
in response to the object name not matching at least one volume identifier of volumes of the composite aggregate, designating the object as an orphan object no longer referenced by at least one volume of the composite aggregate; and
issuing a delete command to the remote object store to delete the orphan object based upon the orphan object being designated as no longer being referenced by the composite aggregate.
7 . The method of claim 1 , wherein the composite aggregate is composed of a plurality of volumes that include the volume, and wherein the method further comprises:
in response to the object name not matching at least one volume identifier of the plurality of volumes, deleting the object from the remote object store.
8 . The method of claim 1 , further comprising:
defining the context of the object page to specify an error indicator as to whether there is an error associated with reading the data chunk to create the object; and
utilizing the context to determine whether to provide access to the data chunk.
9 . The method of claim 1 , wherein the composite aggregate is composed of a plurality of volumes that include the volume, and wherein the method further comprises:
in response to the object name not matching at least one volume identifier of the plurality of volumes of the composite aggregate, designating the object as an orphan object no longer referenced by at least one volume of the composite aggregate;
evaluating the object name of the orphan object to identify orphan volume identifiers of volumes to which the object previously belonged; and
providing the orphan volume identifiers to a user for selective deletion of the orphan object.
10 . The method of claim 1 , comprising:
obtaining a list of volume identifiers of volumes with which the composite aggregate is composed;
transmitting a request to the remote object store for a list of object names of objects stored by the remote object store, wherein the list of object names includes the object name of the object;
comparing the list of volume identifiers with the list of object names;
in response to the object name of the object matching at least one volume identifier within the list of volume identifiers, retaining the object within the remote object store; and
in response to the object name of the object not matching at least one volume identifier within the list of volume identifiers, deleting the object from the remote object store.
11 . A computing device comprising:
a memory comprising machine executable code; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the computing device to:
determine that a data chunk, of a volume, is to be moved from a storage tier to a remote object store, wherein the volume is part of a composite aggregate of data stored across the storage tier and the remote object store;
create an object comprising an object header and an object page for storing the data chunk, wherein the object is created by:
generating an object name for the object based upon a volume identifier of the volume referencing the data chunk;
populating the object header with an identifier of the object name;
populating the object page with the data chunk and a context associated with the object page; and
storing the object into the remote object store; and
compare the object name, determined from the identifier within the object header, with the volume identifier of the volume to determine whether the object name and volume identifier match;
in response to the object name and volume identifier matching, provide access to the data chunk according to the context associated with the object page; and
in response to the object name and volume identifier not matching, determine that the object is no longer referenced by the volume.
12 . The computing device of claim 11 , wherein the machine executable code cause the computing device to:
in response to the object name not matching at least one volume identifier of volumes of the composite aggregate, designate the object as an orphan object no longer referenced by at least one volume of the composite aggregate; and
delete the orphan object based upon the orphan object being designated as no longer being referenced by the composite aggregate.
13 . The computing device of claim 11 , wherein the machine executable code cause the computing device to:
in response to the object name not matching at least one volume identifier of volumes of the composite aggregate, designate the object as an orphan object no longer referenced by at least one volume of the composite aggregate;
evaluate the object name of the orphan object to identify orphan volume identifiers of volumes to which the object previously belonged; and
provide the orphan volume identifiers to a user for deleting the orphan object from the remote object store.
14 . The computing device of claim 11 , wherein the machine executable code cause the computing device to:
obtain a list of volume identifiers of volumes with which the composite aggregate is composed;
transmit a request to the remote object store for a list of object names of objects stored by the remote object store, wherein the list of object names includes the object name of the object;
compare the list of volume identifiers with the list of object names;
in response to the object name of the object matching at least one volume identifier within the list of volume identifiers, retain the object within the remote object store; and
in response to the object name of the object not matching at least one volume identifier within the list of volume identifiers, delete the object from the remote object store.
15 . The computing device of claim 11 , wherein the machine executable code cause the computing device to:
generate the identifier of the object name based upon a hash of a buff tree universal identifier used as the volume identifier; and
in response to storing the object into the remote object store, read the hash back from the object within the remote object store to verify the object.
16 . The computing device of claim 11 , wherein the machine executable code cause the computing device to:
assign the object name based upon the volume identifier and a sequence number assigned to the object for referencing the object; and
utilize the sequence number within the object name to validate a metafile used by a storage system to track objects, within the remote object store, storing data of the volume.
17 . A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:
determine that a data chunk, of a volume, is to be moved from a storage tier to a remote object store;
create an object comprising an object header and one or more an object page for storing the data chunk, wherein the object is created by:
generating an object name for the object based upon a volume identifier of the volume referencing the data chunk;
populating the object header with an identifier of the object name;
populating the object page with the data chunk; and
storing the object into the remote object store; and
compare the object name, determined from the identifier within the object header, with the volume identifier of the volume to determine whether the object name and volume identifier match;
in response to the object name and volume identifier matching, provide access to the data chunk; and
in response to the object name and volume identifier not matching, determine that the object is no longer referenced by the volume.
18 . The non-transitory machine readable medium of claim 17 , wherein the instructions cause the machine to:
assign the object name based the volume identifier and a sequence number assigned to the object for referencing the object; and
utilize the sequence number within the object name to validate a metafile used by a storage system to track objects, within the remote object store, storing data of the volume.
19 . The non-transitory machine readable medium of claim 17 , wherein the instructions cause the machine to:
create a hash for the object storing the data chunk, wherein the hash is created from both the volume identifier of the volume referencing the data chunk and a sequence number assigned to the object for referencing the object;
derive the object name for the object based upon the hash; and
verify the object name of the object by reading the hash from the object within the remote object store.
20 . The non-transitory machine readable medium of claim 17 , wherein the instructions cause the machine to:
generate the identifier of the object name based upon a hash of a buff tree universal identifier used as the volume identifier; and
in response to storing the object into the remote object store, read the hash back from the object within the remote object store to verify the object.