Implementation for efficient log storage
Methods and systems of a log engine for logging data operation in a file system is disclosed. The log engine includes a user interface module configured to receive a plurality of data operation logs that respectively record a data operation and a sequence ID corresponding to an order of the data operation being performed; and a shared log manager configured to provide an in queue configured to consolidate the plurality of data operation logs, and swap the plurality of data operation logs to an out queue with the plurality of data operation logs being ordered based on the sequence ID and saved in a shared log.
1. A log engine configured to log data operation in a file system, the log engine comprising:
a physical processor configured to execute instructions to generate:
a user interface module configured to receive a plurality of data operation logs that respectively record a data operation and a sequence ID corresponding to an order of the data operation being performed; and
a shared log manager, implemented with a lock-free queue, that is configured to:
provide an in queue configured to consolidate the plurality of data operation logs, and
swap the plurality of data operation logs to an out queue with the plurality of data operation logs being ordered based on the sequence ID and saved in memory in a shared log, wherein
the shared log includes the plurality of data operation logs in sequential order based on the sequence ID and extended memory allocations such that tail latency from random write is avoided, wherein the extended memory allocations result from an extend of memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.
2. The log engine of claim 1 , wherein the processor is configured to further generate:
a key-value file system collaborator configured to persist the shared log in the predefined file size allocation for updating the shared log; and
wherein the shared log manager is further configured to request the extend of memory allocation for storing the shared log when the shared log having the file size exceeds the predefined file size allocation.
3. The log engine of claim 1 , wherein the processor is configured to further generate:
when saving the shared log in a persistent storage, a key-value file system collaborator configured to omit metadata on the file size of the shared log.
4. The log engine of claim 1 , wherein
the shared log manager is configured to conduct a cyclic redundancy check for error detection in a log file.
5. The log engine of claim 1 , wherein the processor is configured to further generate:
a background thread configured to move a log file of a metadata log file to persistent storage when the metadata log file exceeds a predefined file size.
6. The log engine of claim 1 , wherein the processor is configured to further generate:
a metadata manager configured to move a metadata log file to persistent storage.
7. The log engine of claim 1 , wherein the processor is configured to further generate:
a garbage collection module of the log engine, the garbage collection module configured to chronologically validate entries in the shared log.
8. The log engine of claim 1 , wherein the processor is configured to further generate:
a recovery manager configured to read a checkpoint file for executing a fuzzy checkpoint to recover from a system failure.
9. A method for logging data operation in a file system, the method comprising:
receiving a plurality of data operation logs that respectively record a data operation and a sequence ID corresponding to an order of the data operation being performed;
consolidating the plurality of data operation logs in a lock-free in queue; and
swapping the plurality of data operation logs to an out queue with the plurality of data operation logs being ordered based on the sequence ID and saved in a shared log that includes the plurality of data operation logs in sequential order based on the sequence ID and extended memory allocations such that tail latency from random write is avoided, wherein the extended memory allocations result from an extend of memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.
10. The method of claim 9 , further comprising:
persisting the shared log in the predefined file size allocation for updating the shared log while omitting metadata on the file size of the shared log; and
requesting the extend of memory allocation for storing the shared log when the shared log having the file size exceeds the predefined file size allocation.
11. The method of claim 9 , further comprising:
conducting a cyclic redundancy check for error detection in a log file.
12. The method of claim 9 , further comprising:
making a log file of a metadata log persistent by a background thread when the log file exceeds a predefined file size.
13. The method of claim 9 , further comprising:
chronologically validating entries in the shared log for garbage collection.
14. The method of claim 9 , further comprising:
reading a checkpoint file for executing a fuzzy checkpoint to recover from a system failure.
15. A non-transitory computer-readable medium containing instructions that, when executed by a processor, direct the processor to:
receive a plurality of data operation logs that respectively record a data operation and a sequence ID corresponding to an order of the data operation being performed;
provide a lock-free in queue configured to consolidate the plurality of data operation logs; and
swap the plurality of data operation logs to an out queue with the plurality of data operation logs being ordered based on the sequence ID and saved in a shared log that includes the plurality of data operation logs in sequential order based on the sequence ID and extended memory allocations such that tail latency from random write is avoided, wherein the extended memory allocations result from an extend of memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.
16. The non-transitory computer-readable medium of claim 15 further containing instructions that, when executed by the processor, direct the processor to:
persist the shared log in the predefined file size allocation for updating the shared log while omitting metadata on the file size of the shared log, and
request the extend of memory allocation for storing the shared log when the shared log having the file size exceeds the predefined file size allocation.
17. The non-transitory computer-readable medium of claim 15 further containing instructions that, when executed by the processor, direct the processor to:
conducting a cyclic redundancy check for error detection in a log file.
18. The non-transitory computer-readable medium of claim 15 further containing instructions that, when executed by the processor, direct the processor to:
make a log file of a metadata log persistent, using a background thread, when the log file exceeds a predefined file size.
19. The non-transitory computer-readable medium of claim 15 further containing instructions that, when executed by the processor, direct the processor to:
chronologically validate entries in the shared log for garbage collection.
20. The non-transitory computer-readable medium of claim 15 further containing instructions that, when executed by the processor, direct the processor to:
read a checkpoint file for executing a fuzzy checkpoint to recover from a system failure.