Chunk retention in a distributed object storage system using stream sessions and stream session backreferences
The present disclosure provides an advantageous systems and related methods for scalable object storage. In accordance with an embodiment of the invention, a stream session is opened to initiate a retention epoch during which a series of edits to an object by a source may be performed without seeing edits from other stream sessions. Backreferences identifying the stream session (i.e. stream session backreferences) are applied to retained chunks. While a payload chunk has one or more such associated stream session backreferences, the chunk is retained and not expunged. Advantageously, transitory chunks that are put during the retention epoch may be unput promptly so as to release valuable cache resources. In comparison, the prior approach only eventually releases the data storage resources for expunged version manifests. Other embodiments, aspects and features are also disclosed.
1. A method for storing chunks of data objects in a distributed object storage system, the method comprising:
encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests; and
encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch;
requesting by the source a put of a version manifest to the system during the retention epoch;
reading chunk references that are contained, directly or indirectly, in the version manifest;
using the chunk references to determine inherited chunks, wherein the inherited chunks are chunks from a reference version manifest that is identified by the stream session; and
performing a touch operation to add stream session backreferences to the inherited chunks.
2. A method for storing chunks of data objects in a distributed object storage system, the method comprising:
encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests; and
encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch;
requesting by the source a put of a version manifest to the system during the retention epoch;
reading chunk references that are contained, directly or indirectly, in the version manifest;
using the chunk references to determine inherited chunks, wherein the inherited chunks are chunks from a reference version manifest that is identified by the stream session; and
using the chunk references to determine put-during-session chunks,
wherein the put-during-session chunks are chunks that were put previously during the retention epoch.
3. The method of claim 2 further comprising:
unputting transitory chunks, which are put-during-session chunks that are not in a most recent version manifest.
4. The method of claim 3 further comprising:
releasing cache resources when the transitory chunks are unput.
5. The method of claim 2 further comprising:
using the chunk references to determine new chunks,
wherein the new chunks were not put previously during the retention epoch and are not inherited chunks.
6. The method of claim 5 further comprising:
putting the new chunks to the system, each new chunk being put with a stream session backreference.
7. A system for storing chunks of data objects, the system comprising:
one or more servers encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests; and
one or more servers encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch,
wherein the source requests a put of a version manifest to the system during the retention epoch, chunk references that are contained, directly or indirectly, in the version manifest are read, and the chunk references are used to determine inherited chunks, the inherited chunks being chunks from a reference version manifest that is identified by the stream session, and
wherein a touch operation is performed to add stream session backreferences to the inherited chunks.
8. A distributed storage system for storing chunks of data objects, the system comprising:
one or more servers encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests; and
one or more servers encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch,
wherein the source requests a put of a version manifest to the system during the retention epoch, chunk references that are contained, directly or indirectly, in the version manifest are read, and the chunk references are used to determine inherited chunks, the inherited chunks being chunks from a reference version manifest that is identified by the stream session, and
wherein the chunk references are used to determine put-during-session chunks, and the put-during-session chunks are chunks that were put previously during the retention epoch.
9. The system of claim 8 , wherein transitory chunks are unput, wherein the transitory chunks are put-during-session chunks that are not in a most recent version manifest.
10. The system of claim 9 , wherein cache resources are released when the transitory chunks are unput.
11. The system of claim 8 , wherein the chunk references are used to determine new chunks, and wherein the new chunks were not put previously during the retention epoch and are not inherited chunks.
12. The system of claim 11 , wherein the new chunks are put to the system, each new chunk being put with a stream session backreference.