Data tiering in a cluster network using distributed shared logging with multiple segment allocation

1 . A method of migrating log data to tiered storage units in a cluster system, comprising:

first deploying a plurality of nodes in the cluster system, each node generating logs for system activity and maintained in a respective client, an associated client library, a respective log server, and a respective storage device comprising non-volatile RAM (NVRAM) devices;

second deploying tiered storage comprising memory devices of lesser cost and lower performance than the NVRAM devices;

storing the logs in a shared log structure that is segmented into a plurality of segments based on the logs and respective entries;

maintaining in one node of the plurality of nodes, a metastore containing a mapping of the logs with respective storage locations in the respective storage devices; and

identifying a segment of the plurality of segments that is sealed against modification or full of data and migrating the segment to the tiered storage based on at least one of time and space threshold values.

2 . The method of claim 1 wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster system comprises part of a Data Domain deduplication backup system performing backup and restore operations for the nodes, and further wherein the non-volatile RAM based storage comprises Non-Volatile Memory Express (NVMe) solid state disk (SSD) storage, and yet further wherein the tiered storage comprises at least one of hard disk drives (HDDs), cloud storage, or long-term archive storage.

3 . The method of claim 2 wherein the time threshold is based on a time-to-live (TTL) parameter specifying how long the segment should be stored in the tiered storage before being automatically deleted or marked for overwrite.

4 . The method of claim 3 wherein the tiered storage comprises an erasure encoded storage unit that eliminates a replica of the segment, and the TTL parameter eliminates the need for a garbage collection process to remove the segment.

5 . The method of claim 2 further comprising:

checking an amount of space occupied by segments in the NVMe SSD;

comparing the amount of space to a defined space threshold; and

migrating, if the space threshold is exceeded, the segment to the tiered storage.

6 . The method of claim 5 further comprising truncating the segment to reclaim storage space when the defined space threshold is reached.

7 . The method of claim 2 further comprising sequencing the logs in segment set including the segment, and that maps each log to a unique location in a respective segment, wherein each segment consists of blocks and each block consists of sequence of entries, and each entry comprises a data blob written by the client, and further wherein the metastore is a stateless service, and maintains through a key value store (KVS), the mapping of all segments available in the cluster network to the log servers which host the segments.

8 . The method of claim 7 further comprising:

assigning a log sequence number (LSN) to uniquely identify a data blob comprising the log upon appending data to the log;

assigning a segment set sequence number (SSN) to each segment in a log;

processing a client log write by appending new log segments to the log;

issuing, from the associated client library an append to log servers on different nodes;

committing, by the log servers, of a log entry to the log segments;

processing a log read operation upon restart of a client service;

issuing the log read operation by a client using a respective client library to read an appropriate log segment; and

performing the log read operation by a log server.

9 . The method of claim 8 further comprising:

writing the segment to the tiered storage;

updating the data mapping in the metastore to indicate the new location of the segment in the tiered storage; and

updating the mapping to clears a previous segment allocation on the log servers including the respective log server.

10 . A method of storing log data in a tiered storage system for a cluster system using a distributed log, comprising:

deploying memory resources comprising non-volatile RAM (NVRAM) and tiered storage comprising memory devices of lesser cost and lower performance than the NVRAM devices;

storing logs of system activity generated by nodes in the cluster system in the NVRAM, wherein the logs are stored in a shared log structure that is segmented into a plurality of segments based on the logs and respective entries;

maintaining in one node of the nodes, a metastore containing a mapping of the logs with respective storage locations in the respective storage devices; and

migrating logs from the NVRAM to the tiered storage based on at least one of time and space thresholds.

11 . The method of claim 10 wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster system comprises part of a Data Domain deduplication backup system performing backup and restore operations for the nodes, and further wherein the non-volatile RAM based storage comprises Non-Volatile Memory Express (NVMe) solid state disk (SSD) storage, and yet further wherein the tiered storage comprises at least one of hard disk drives (HDDs), cloud storage, or long-term archive storage.

12 . The method of claim 11 wherein the time threshold is based on a time-to-live (TTL) parameter specifying how long the segment should be stored in the tiered storage before being automatically deleted or marked for overwrite, and further wherein the space threshold is based on an amount of segment data stored in the NVMe SSD storage.

13 . The method of claim 12 wherein the tiered storage comprises an erasure encoded storage unit that eliminates a replica of the segment, and the TTL parameter eliminates the need for a garbage collection process to remove the segment.

14 . The method of claim 12 further comprising truncating the segment to reclaim storage space when the defined space threshold is reached.

15 . The method of claim 14 further comprising sequencing the logs in segment set including the segment, and that maps each log to a unique location in a respective segment, wherein each segment consists of blocks and each block consists of sequence of entries, and each entry comprises a data blob written by the client, and further wherein the metastore is a stateless service, and maintains through a key value store (KVS), the mapping of all segments available in the cluster network to log servers which host the segments.

16 . The method of claim 14 further comprising:

writing the segment to the tiered storage;

updating the data mapping in the metastore to indicate the new location of the segment in the tiered storage; and

updating the mapping to clears a previous segment allocation on the log servers including the log servers.

17 . A system migrating log data to tiered storage units in a cluster system, comprising:

a plurality of nodes, wherein each node generates logs for system activity and maintained in a respective client, an associated client library, a respective log server, and a respective storage device comprising non-volatile RAM (NVRAM) devices;

tiered storage comprising memory devices of lesser cost and lower performance than the NVRAM devices;

a shared log structure maintained in storage and storing the logs in a shared log structure that is segmented based on the logs and respective entries;

a metastore component maintaining in one node of the plurality of nodes, a mapping of the logs with respective storage locations in the respective storage devices; and

a tiering component identifying a segment of the plurality of segments that is sealed against modification or full of data and migrating the segment to the tiered storage based on at least one of time and space threshold values.

18 . The system of claim 17 wherein the cluster system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the cluster system comprises part of a Data Domain deduplication backup system performing backup and restore operations for the nodes, and further wherein the non-volatile RAM based storage comprises Non-Volatile Memory Express (NVMe) solid state disk (SSD) storage, and yet further wherein the tiered storage comprises at least one of hard disk drives (HDDs), cloud storage, or long-term archive storage.

19 . The system of claim 18 wherein the time threshold is based on a time-to-live (TTL) parameter specifying how long the segment should be stored in the tiered storage before being automatically deleted or marked for overwrite, and further wherein the space threshold is based on an amount of segment data stored in the NVMe SSD storage.

20 . The system of claim 19 further comprising a truncating component the segment to reclaim storage space when the defined space threshold is reached.