Multiple file compaction for network attached storage
Mechanisms are provided for optimizing multiple files in an efficient format that allows maintenance of the original namespace. Multiple files and associated metadata are written to a suitcase file. The suitcase file includes index information for accessing compressed data associated with compacted files. A hardlink to the suitcase file includes an index number used to access the appropriate index information. A simulated link to a particular file maintains the name of the particular file prior to compaction.
1. A system comprising:
a processor operable to create a suitcase file operable to hold a plurality of files, the processor operable to compress and compact the plurality of files for inclusion in the suitcase file, wherein each of the plurality of files is compressed to increase the storage efficiency of each of the plurality of files and wherein the plurality of files are compacted as a group of files to increase storage efficiency of the group of files, wherein the processor performs deduplication to compact the plurality of files;
storage operable to maintain the suitcase file, the suitcase file having a plurality of metadata for each of the plurality of files, wherein for each of the plurality of metadata of the plurality of files:
a first copy of metadata is surrounded by a first commit bit and a second commit bit and a second copy of the metadata is surrounded by the second commit bit and a third commit bit, wherein if any of the first, second, and third bits surrounding the first copy and the second copy of the metadata has not been written, the metadata is not considered to be committed, wherein if all of the first, second, and third bits surrounding the first copy and the second copy of the metadata have been written, the metadata is considered to be accurately written;
wherein the processor is further operable to access the plurality of files in the suitcase file using index information included in the suitcase file.
2. The system of claim 1 , wherein an index number in a file name is used to access index information in the suitcase file to allow access to corresponding compressed and compacted data in the suitcase file.
3. The system of claim 1 , wherein commit bits are written immediately before and immediately after metadata is written.
4. The system of claim 1 , wherein index information comprises a next_index and a next_offset.
5. The system of claim 1 , wherein index information further comprises data_length and data_info fields for each of the plurality of files.
6. The system of claim 1 , wherein file name index number are included in each of the hard links to the plurality of files in the suitcase file.
7. The system of claim 1 , wherein a simulated link is mapped to each of the hard links.
8. The system of claim 7 , wherein the simulated link to a compressed and compacted file has a name matching the name of the file in precompressed and precompacted form.
9. A method comprising:
creating a suitcase file operable to hold a plurality of files;
compressing each of the plurality of files individual to increase storage efficiency of each of the plurality of files;
compacting the plurality of files as a group to increase storage efficiency of the group of files, wherein compacting the plurality of files includes deduplication;
writing a plurality of metadata for each of the plurality of files into the suitcase file and onto storage, wherein for each of the plurality of metadata of the plurality of files, writing metadata comprises sequentially writing a first commit bit, a first copy of metadata for a file, a second commit bit, a second copy of metadata for the file, and a third commit bit, wherein a first copy of metadata is surrounded by a first commit bit and a second commit bit and a second copy of the metadata is surrounded by the second commit bit and a third commit bit, wherein if any of the first, second, and third bits surrounding the first copy and the second copy of the metadata has not been written, the metadata is not considered to be committed, wherein if all of the first, second, and third bits surrounding the first copy and the second copy of the metadata have been written, the metadata is considered to be accurately written;
accessing the plurality of files in the suitcase file using index information included in the suitcase file.
10. The method of claim 9 , wherein an index number in a file name is used to access index information in the suitcase file to allow access to corresponding compressed and compacted data in the suitcase file.
11. The method of claim 9 , wherein commit bits are written immediately before and immediately after metadata is written.
12. The method of claim 9 , wherein index information comprises a next_index and a next_offset.
13. The method of claim 9 , wherein index information further comprises data_length and data_info fields for each of the plurality of files.
14. The method of claim 9 , wherein file name index number are included in each of the hard links to the plurality of files in the suitcase file.
15. The method of claim 9 , wherein a simulated link is mapped to each of the hard links.
16. The method of claim 15 , wherein the simulated link to a compressed and compacted file has a name matching the name of the file in precompressed and precompacted form.
17. A non-transitory computer readable medium comprising:
computer code for creating a suitcase file operable to hold a plurality of files;
computer code for compressing each of the plurality of files individual to increase storage efficiency of each of the plurality of files;
computer code for compacting the plurality of files as a group to increase storage efficiency of the group of files, wherein compacting the plurality of files includes deduplication;
computer code for writing a plurality of metadata for each of the plurality of files into the suitcase file and onto storage, wherein for each of the plurality of metadata of the plurality of files, writing metadata comprises sequentially writing a first commit bit, a first copy of metadata for a file, a second commit bit, a second copy of metadata for the file, and a third commit bit, wherein a first copy of metadata is surrounded by a first commit bit and a second commit bit and a second copy of the metadata is surrounded by the second commit bit and a third commit bit, wherein if any of the first, second, and third bits surrounding the first copy and the second copy of the metadata has not been written, the metadata is not considered to be committed, wherein if all of the first, second, and third bits surrounding the first copy and the second copy of the metadata have been written, the metadata is considered to be accurately written;
computer code for accessing the plurality of files in the suitcase file using index information included in the suitcase file.
18. The non-transitory computer readable medium of claim 17 , wherein an index number in a file name is used to access index information in the suitcase file to allow access to corresponding compressed and compacted data in the suitcase file.
19. The non-transitory computer readable medium of claim 17 , wherein commit bits are written immediately before and immediately after metadata is written.
20. The non-transitory computer readable medium of claim 17 , wherein index information comprises a next_index and a next_offset.