Systems and methods for scalable object storage
The present disclosure provides an advantageous systems and related methods for scalable object storage. One embodiment relates to a system that includes at least a manifest tracking subsystem and a chunk payload subsystem. The manifest tracking subsystem stores version manifests and encodes forward-references from the version manifests to chunks. The chunk payload subsystem stores the chunks and encodes back-references to version manifests that reference the chunk. Other embodiments, aspects and features are also disclosed.
1. A system for providing distributed storage of data objects, the system comprising:
a manifest tracking subsystem comprising a plurality of manifest server computers communicatively interconnected by a network, the manifest tracking subsystem encoding forward-references from version manifests to chunks, wherein each forward-reference references a chunk by an identity of the chunk without identifying any server on which the chunk is stored; and
a chunk payload subsystem comprising a plurality of chunk server computers communicatively interconnected by the network, the chunk payload subsystem storing the chunks and encodes back-references to version manifests that reference the chunk, wherein each back-reference references an object by an identity of the object without identifying any server on which the object is stored,
further wherein the back-references maintained by the chunk payload subsystem comprise speculative back-references and verified back-references,
further wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and the speculative back-reference is deleted if the speculative back-reference is of a lower class-of-storage than a verified back-reference from the chunk to the object.
2. The system of claim 1 , wherein the speculative back-reference encodes a transaction identifier supplied with the chunk put operation.
3. The system of claim 1 , wherein a put cancelling operation that identifies the chunk and the transaction identifier causes deletion of the speculative back-reference.
4. The system of claim 1 , wherein the speculative back-reference further encodes a timestamp of the object put transaction.
5. The system of claim 1 , wherein the speculative back-reference is created even when the chunk previously existed in the chunk payload subsystem.
6. The system of claim 1 , wherein a verified back-reference corresponding to the speculative back-reference is created only after the manifest tracking subsystem has completed a transaction that included the chunk put operation.
7. The system of claim 1 , wherein, in order to verify the speculative back-reference, the chunk payload subsystem queries the manifest tracking subsystem to confirm that there is a corresponding forward-reference to the chunk from the object.
8. The system of claim 1 further comprising:
a client which initiates an object put transaction,
wherein the client obtains a transaction identifier for the object put transaction from the manifest tracking subsystem by sending a request for transaction identifier that at least includes an identity of the object to be put.
9. The system of claim 8 , wherein the request for transaction identifier includes a class-of-storage identifier.
10. The system of claim 8 , wherein the manifest tracking subsystem cryptographically signs the transaction identifier, and wherein the transaction identifier encodes at least the identity of the object to be put and a unique transaction identifier.
11. The system of claim 10 , wherein the transaction identifier is unique even when there are multiple concurrent transactions updating the object to be put.
12. The system of claim 8 , wherein the chunk payload subsystem rejects a request for a chunk put operation if the request does not supply a transaction identifier which was cryptographically signed by the manifest subsystem.
13. The system of claim 1 , wherein the manifest tracking subsystem and the chunk payload subsystem collaborate to maintain a class-of-storage attribute for each chunk stored in the system.
14. The system of claim 13 , wherein the manifest tracking subsystem issues a notification of a class-of-storage update when a class-of-storage required for a chunk by an object has been updated.
15. The system of claim 14 , wherein the chunk payload subsystem applies the class-of-storage update to a verified back-reference.
16. The system of claim 14 , wherein the chunk payload subsystem creates the verified back-reference, if necessary, by confirming a speculative back-reference between the chunk and the object.
17. The system of claim 1 , wherein a chunk get operation is validated using a transaction identifier obtained from the manifest tracking subsystem and supplied to the chunk payload subsystem with the chunk get operation.
18. The system of claim 17 , wherein the chunk payload subsystem rejects the chunk get operation if the chunk does not have a back-reference to an object identified in the transaction identifier.
19. A system for providing distributed storage of data objects, the system comprising:
a manifest tracking subsystem comprising a plurality of manifest server computers communicatively interconnected by a network, the manifest tracking subsystem encoding forward-references from version manifests to chunks, wherein each forward-reference references a chunk by an identity of the chunk without identifying any server on which the chunk is stored; and
a chunk payload subsystem comprising a plurality of chunk server computers communicatively interconnected by the network, the chunk payload subsystem storing the chunks and encoding back-references to version manifests that reference the chunk, wherein each back-reference references an object by an identity of the object without identifying any server on which the object is stored,
further wherein the back-references maintained by the chunk payload subsystem comprise speculative back-references and verified back-references,
further wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and, in order to verify the speculative back-reference, the chunk payload subsystem queries the manifest tracking subsystem to confirm that there is a corresponding forward-reference to the chunk from the object,
further wherein upon confirmation of the corresponding forward-reference, a verified back-reference from the chunk to the object is created if none already exists, and a class-of-storage attribute for the verified back-reference is updated if the verified back-reference already exists,
further wherein the speculative back-reference is then deleted.
20. The system of claim 19 , wherein the class-of-storage attribute is updated to be a greater of the class-of-storage attributes in the speculative and the verified back-references.
21. A system for providing distributed storage of data objects, the system comprising:
a manifest tracking subsystem comprising a plurality of manifest server computers interconnected by a network, the manifest tracking subsystem encoding forward-references from version manifests to chunks, wherein each forward-reference references a chunk by an identity of the chunk without identifying any server on which the chunk is stored; and
a chunk payload subsystem comprising a plurality of chunk server computers interconnected by the network, the chunk payload subsystem storing the chunks and encoding back-references to version manifests that reference the chunk, wherein each back-reference references an object by an identity of the object without identifying any server on which the object is stored,
further wherein the back-references maintained by the chunk payload subsystem comprise speculative back-references and verified back-references,
further wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and, in order to verify the speculative back-reference, the chunk payload subsystem queries the manifest tracking subsystem to confirm that there is a corresponding forward-reference to the chunk from the object,
further wherein upon confirmation of the corresponding forward-reference, a reference count for the chunk for the specific class-of-storage is incremented,
further wherein the speculative back-reference is then deleted.
22. The system of claim 21 , wherein reference counts are adjusted in response to notifications from the manifest tracking subsystem that a class-of-storage for the chunk from a specific object has changed.
23. A method for providing storage of data objects using a distributed storage system, the method comprising:
encoding forward-references from version manifests to chunks in a manifest tracking subsystem which stores the version manifests, wherein each forward-reference references a chunk by an identity of the chunk without identifying any server on which the chunk is stored; and
encoding back-references to version manifests that reference the chunk, wherein each back-reference references an object by an identity of the object without identifying any server on which the object is stored,
further wherein the back-references maintained by the chunk payload system comprise speculative back-references and verified back-references,
further wherein a speculative back-reference is created during a chunk put operation to put a chunk into the system, and the speculative back-reference is deleted if the speculative back-reference is of a lower class-of-storage than a verified back-reference from the chunk to the object.