Metadata compression
A computer-executable method, system, and computer program product for managing a Parallel-Logged File System (PLFS) on a data storage system, wherein the data storage system is in communication with an application that updates the data storage system with checkpoint data, the computer-executable method, system, and computer program product comprising receiving a first checkpoint including data from the application; analyzing the data in the first checkpoint to determine whether a pattern exists in the checkpoint data; and upon a positive determination, adding the pattern into PLFS.
1. A computer-executable method of managing a Parallel-Logged File System (PLFS) on a data storage system, wherein the data storage system is in communication with an application that updates the data storage system with checkpoint data, the computer-executable method comprising:
receiving a first checkpoint including the checkpoint data from the application;
analyzing the checkpoint data in the first checkpoint to determine whether a pattern exists in the checkpoint data, wherein the analyzing comprises:
pre-processing the checkpoint data, using a discovery engine within the data storage system, by creating a list of deltas from the checkpoint data, wherein the deltas are calculated between respective pairs of offsets of a plurality of offsets recorded for a plurality of writes for one or more compute nodes;
analyzing the list of deltas by using a look-ahead window to determine whether a pattern exists;
wherein the look-ahead window comprises a size accommodating less than all of the deltas;
wherein using the look-ahead window to determine whether a pattern exists comprises incrementally moving the look-ahead window along the deltas to find one or more deltas in a repeating sequence; and
upon a positive determination, adding the pattern as a corresponding pattern entry into the PLFS; and
consolidating multiple patterns within the PLFS, wherein the consolidated multiple patterns reduce a size of a PLFS metadata index;
wherein the adding comprises creating the corresponding pattern entry; and
wherein the corresponding pattern entry indicates the one or more deltas in the repeating sequence and a corresponding number of each of the repetitions of the repeating sequence.
2. The computer-executable method of claim 1 , wherein the adding further comprises:
inserting the corresponding pattern entry in the PLFS metadata index.
3. The computer-executable method of claim 2 , further comprising:
analyzing the PLFS metadata index for a global pattern; and
creating a second corresponding pattern entry for the global pattern; and
removing one or more entries from the PLFS metadata index associated with the global pattern; and
representing the removed one or more entries with the second corresponding pattern entry in the PLFS metadata index.
4. The computer-executable method of claim 3 , wherein the global pattern is a collection of repeating patterns within the PLFS metadata index.
5. The computer-executable method of claim 1 , wherein the corresponding pattern entry represents a plurality of single entries.
6. The computer-executable method of claim 1 , wherein the analyzing of the list of deltas comprises using a search window in addition to the look-ahead window to determine whether a pattern exists.
7. The computer-executable method of claim 6 , wherein using the search window to determine whether a pattern exists comprises incrementally moving the search window with the look-ahead window along the deltas to find the one or more deltas in the repeating sequence.
8. A system, comprising:
a host system running an application;
a data storage system running a Parallel-Logged File system (PLFS), wherein the data storage system is in communication with the application that updates the data storage system with checkpoint data; and
computer-executable logic encoded in memory of one or more computers in communication with the data storage system to enable the management of the PLFS on the data storage system, wherein the computer-executable program logic is configured for the execution of:
receiving a first checkpoint including the checkpoint data from the application;
analyzing the checkpoint data in the first checkpoint to determine whether a pattern exists in the checkpoint data, wherein the analyzing comprises:
pre-processing the checkpoint data, using a discovery engine within the data storage system, by creating a list of deltas from the checkpoint data, wherein the deltas are calculated between respective pairs of offsets of a plurality of offsets recorded for a plurality of writes for one or more compute nodes;
analyzing the list of deltas by using a look-ahead window to determine whether a pattern exists;
wherein the look-ahead window comprises a size accommodating less than all of the deltas;
wherein using the look-ahead window to determine whether a pattern exists comprises incrementally moving the look-ahead window along the deltas to find one or more deltas in a repeating sequence; and
upon a positive determination, adding the pattern as a corresponding pattern entry into the PLFS; and
consolidating multiple patterns within the PLFS, wherein the consolidated multiple patterns reduce a size of a PLFS metadata index;
wherein the adding comprises creating the corresponding pattern entry; and
wherein the corresponding pattern entry indicates the one or more deltas in the repeating sequence and a corresponding number of each of the repetitions of the repeating sequence.
9. The system of claim 8 , wherein the adding further comprises:
inserting the corresponding pattern entry in the PLFS metadata index.
10. The system of claim 9 , wherein the computer-executable program logic is further configured for the execution of:
analyzing the PLFS metadata index for a global pattern; and
creating a second corresponding pattern entry for the global pattern; and
removing one or more entries from the PLFS metadata index associated with the global pattern; and
representing the removed one or more entries with the second corresponding pattern entry in the PLFS metadata index.
11. The system of claim 10 , wherein the global pattern is a collection of repeating patterns within the PLFS metadata index.
12. The system of claim 8 , wherein the corresponding pattern entry represents a plurality of single entries.
13. The system of claim 8 , wherein the analyzing of the list of deltas comprises using a search window in addition to the look-ahead window to determine whether a pattern exists.
14. The system of claim 13 , wherein using the search window to determine whether a pattern exists comprises incrementally moving the search window with the look-ahead window along the deltas to find the one or more deltas in the repeating sequence.
15. A computer program product for managing a Parallel-Logged File System (PLFS) on a data storage system, wherein the data storage system is in communication with an application that updates the data storage system with checkpoint data, the computer program product comprising:
a non-transitory computer readable medium encoded with computer-executable program code configured to enable the execution of:
receiving a first checkpoint including the checkpoint data from the application;
analyzing the checkpoint data in the first checkpoint to determine whether a pattern exists in the checkpoint data, wherein the analyzing comprises:
pre-processing the checkpoint data, using a discovery engine within the data storage system, by creating a list of deltas from the checkpoint data, wherein the deltas are calculated between respective pairs of offsets of a plurality of offsets recorded for a plurality of writes for one or more compute nodes;
analyzing the list of deltas by using a look-ahead window to determine whether a pattern exists;
wherein the look-ahead window comprises a size accommodating less than all of the deltas;
wherein using the look-ahead window to determine whether a pattern exists comprises incrementally moving the look-ahead window along the deltas to find one or more deltas in a repeating sequence; and
upon a positive determination, adding the pattern as a corresponding pattern entry into the PLFS; and
consolidating multiple patterns within the PLFS, wherein the consolidated multiple patterns reduce a size of a PLFS metadata index;
wherein the adding comprises creating the corresponding pattern entry; and
wherein the corresponding pattern entry indicates the one or more deltas in the repeating sequence and a corresponding number of each of the repetitions of the repeating sequence.
16. The computer program product of claim 15 , wherein the adding further comprises:
inserting the corresponding pattern entry in the PLFS metadata index.
17. The computer program product of claim 16 , wherein the code is further configured to enable the execution of:
analyzing the PLFS metadata index for a global pattern; and
creating a second corresponding pattern entry for the global pattern; and
removing one or more entries from the PLFS metadata index associated with the global pattern; and
representing the removed one or more entries with the second corresponding pattern entry in the PLFS metadata index.
18. The computer program product of claim 17 , wherein the global pattern is a collection of repeating patterns within the PLFS metadata index.
19. The computer program product of claim 15 , wherein the analyzing of the list of deltas comprises using a search window in addition to the look-ahead window to determine whether a pattern exists.
20. The computer program product of claim 19 , wherein using the search window to determine whether a pattern exists comprises incrementally moving the search window with the look-ahead window along the deltas to find the one or more deltas in the repeating sequence.