Parallel transparent restructuring of immutable content in a distributed object storage system
The present disclosure provides an innovative technique for parallel transparent restructuring of immutable content in a distributed object storage system. The content restructuring technique disclosed herein may be implemented with parallel operations by multiple storage servers in a live system. The restructuring is transparent in that the original CHIT may still be used to retrieve and validate the original content, such that a client requesting the original content need not be aware that the original content has been restructured. Furthermore, the restructuring is performed while maintaining support for pre-existing immutable metadata. Other embodiments, aspects and features are also disclosed. The content remains immutable from the perspective of the end user, but still may be advantageously restructured to optimize storage efficiency.
1. A method of restructuring content stored at a storage server of a distributed object storage system, the method comprising:
determining that an original chunk containing original content is to be restructured into an alternate chunk containing alternate content deriving the alternate content from the original content;
writing the alternate chunk; and
writing an alternate-index key-value tuple (KVT) that establishes the alternate chunk as containing an alternate encoding of the original chunk.
2. The method of claim 1 , wherein the alternate-index KVT comprises:
a searchable key that includes an original chunk identifying token (CHIT) which identifies the original content; and
a value that includes an alternate CHIT which identifies the alternate content.
3. The method of claim 1 , wherein the original content is of a category from a group of categories consisting of a payload category, a Content Manifest category, and a Version Manifest category.
4. The method of claim 1 , wherein the alternate content is of a category from a group of categories consisting of a payload category, a Content Manifest category, and a Version Manifest category.
5. The method of claim 1 , wherein the alternate content, when represented by an alternate manifest, comprises payload chunks using different chunk boundaries; the method comprising:
generating alternate chunks with new boundaries;
creating an alternate manifest KVT to an original manifest that represents the original content; and
issuing verified back references from the original manifest to the alternate chunks referenced using the alternate manifest.
6. The method of claim 1 , wherein the alternate content comprises a compressed version of the original content.
7. The method of claim 1 , wherein the alternate content comprises a slice of a set of erasure-encoded slices generated from the original content.
8. The method of claim 1 , wherein the alternate content comprises changes relative to a content of a referenced chunk.
9. The method of claim 1 , further comprising:
reducing a retention requirement for the original content after successfully writing the alternate chunk and the alternate-index KVT.
10. The method of claim 9 , wherein the retention requirement for a given replica of a chunk on a specific storage server is reduced by marking an original KVT for the original chunk as releasable.