Metadata store in multiple reusable append logs
View Patent ↗A storage system ( 100 ) stores metadata using an append log to provide speed and reliability. Appending entities to the end of a metadata store ( 154 ) and provides reliability in the event of failure because a failed append operation at worst corrupts the end of the append log, which is easily detected and corrected. A metadata store ( 154 ) can be split into sections ( 158 ). A used section ( 158 ) may be identified as garbage when the stored metadata in other sections ( 158 ) make the all of its stored entity sets stale. A used section ( 158 ) can be made garbage by storing the entity sets from the section ( 158 ) in another section ( 158 ). Sections ( 158 ) containing garbage thereby change to unused allow replacement and removal of previous sections ( 158 ) of metadata.
1. A process for storing metadata for a storage system, comprising:
partitioning a data storage device into a plurality sections that are initially unused and are dedicated to an append log for metadata;
splitting first metadata into a first set of entities;
encoding the first set of entities as encoded entities suitable for the append log;
storing the encoded entities in the append log beginning at a first of the sections so that the first section used;
splitting following metadata into a next set of entities;
encoding the next set of entities as next encoded entities suitable for the append log; and
storing the next set of encoded entities in the append log following a prior end of the append log.
2. The process of claim 1 , further comprising:
detecting one of the sections that is used and contains only stale entities; and
designating the detected section as being unused.
3. The process of claim 2 , further comprising:
monitoring numbers of entities that are not stale in a targeted one of the sections; and
in response to the number falling below a limit, storing the entities that are not stale from the target section at the end of the append log so that all of the entities in the target section are stale.
4. The process of claim 1 , further comprising:
monitoring numbers of entities that are not stale in a targeted one of the sections; and
in response to the number falling below a limit, storing the entities that are not stale from the target section at the end of the append log so that all of the entities in the target section are stale.
5. The process of claim 1 , wherein the append log corresponds to a raw partition of the storage device.
6. The process of claim 1 , further comprising a plurality of repetitions of:
splitting following metadata into a next set of entities;
encoding the next set of entities as next encoded entities suitable for the append log; and
storing the next set of encoded entities in the append log following a prior end of the append log.
7. The process of claim 1 , wherein the encoding comprises encoding using JavaScript Object Notation (JSON).
8. A storage processing system configured to control a data storage device and further configured to perform a process including:
partitioning the data storage device into a plurality sections that are initially unused and are dedicated to an append log for metadata that the storage processing system uses;
identifying a condition or operation that affects the metadata that the storage processing system uses;
encoding the affected metadata as a first set of entities suitable for the append log;
storing the encoded entities in the append log beginning at a first of the sections so that the first section used;
identifying a next condition or operation that affects the metadata that the storage processing system uses;
encoding the metadata affected by the next condition or operation as a next set of entities suitable for the append log; and
storing the next set of encoded entities in the append log following a prior end of the append log.
9. The storage processing system of claim 8 , further comprising one or more repetitions of:
identifying a next condition or operation that affects the metadata that the storage processing system uses;
encoding the metadata affected by the next condition or operation as a next set of entities suitable for the append log; and
storing the next set of encoded entities in the append log following a prior end of the append log.