Large content file optimization
A size associated with a content file is determined to be greater than a threshold size. Contents of the content file split across a plurality of component files are stored. Metadata, for the content file, is updated to reference a plurality of component file metadata structures for the component files. A node of the metadata is configured to track different sizes of portions of the content file stored in different component files of the plurality of component files. File metadata of the content file is split across the plurality of component file metadata structures and each component file metadata structure of the plurality of component file metadata structures specifies a corresponding structure organizing data components for a corresponding portion of the content file.
1. A method, comprising:
determining that a size associated with a content file is greater than a threshold size;
storing contents of the content file split across a plurality of component files, wherein a component file of the component files is configured to store a portion of the contents of the content file;
generating a plurality of component file metadata structures for the plurality of component files of the content file, wherein a component file metadata structure of the plurality of component file metadata structures corresponds to one of the plurality of component files of the content file, wherein each of the plurality of component file metadata structures includes a corresponding root node; and
updating metadata, for the content file, to reference the plurality of component file metadata structures for the plurality of component files, wherein a node of the metadata is configured to track different sizes of portions of the content file stored in different component files of the plurality of component files, wherein file metadata of the content file is split across the plurality of component file metadata structures and each component file metadata structure of the plurality of component file metadata structures specifies a corresponding structure organizing data components for a corresponding portion of the content file.
2. The method of claim 1 , wherein the node stores information, wherein the information includes a first vector that indicates corresponding size information associated with each of the plurality of component file metadata structures.
3. The method of claim 2 , wherein the information includes a second vector that indicates a number of sequential component file metadata structures having a corresponding size indicated by the first vector.
4. The method of claim 3 , wherein at least two sequential component file metadata structures are associated with a same portion size of the content file.
5. The method of claim 1 , further comprising receiving a request for file data associated with the content file.
6. The method of claim 5 , further comprising inspecting the node to identify the plurality of component file metadata structures associated with the content file.
7. The method of claim 6 , further comprising:
traversing the plurality of identified component file metadata structures associated with the content file; and
providing the requested file data associated with the content file.
8. The method of claim 1 , further comprising receiving the plurality of component files.
9. The method of claim 8 , wherein at least two of the plurality of component files have a same size.
10. The method of claim 8 , wherein at least two of the plurality of component files have a different size.
11. The method of claim 8 , further comprising generating the plurality of component file metadata structures for the plurality of component files.
12. The method of claim 1 , wherein file offsets associated with the content file are brick aligned.
13. The method of claim 1 , wherein the node is a directory inode.
14. The method of claim 1 , wherein the node includes pointers to two or more other nodes, wherein the two or more other nodes include corresponding pointers to the plurality of component file metadata structures associated with the content file.
15. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
determining that a size associated with a content file is greater than a threshold size;
storing contents of the content file split across a plurality of component files, wherein a component file of the component files is configured to store a portion of the contents of the content file;
generating a plurality of component file metadata structures for the plurality of component files of the content file, wherein a component file metadata structure of the plurality of component file metadata structures corresponds to one of the plurality of component files of the content file, wherein each of the plurality of component file metadata structures includes a corresponding root node; and
updating metadata, for the content file, to reference the plurality of component file metadata structures for the plurality of component files, wherein a node of the metadata is configured to track different sizes of portions of the content file stored in different component files of the plurality of component files, wherein file metadata of the content file is split across the plurality of component file metadata structures and each component file metadata structure of the plurality of component file metadata structures specifies a corresponding structure organizing data components for a corresponding portion of the content file.
16. The computer program product of claim 15 , wherein the node stores information, wherein the information includes a first vector that indicates corresponding size information associated with each of the plurality of component file metadata structures.
17. The computer program product of claim 16 , wherein the information includes a second vector that indicates a number of sequential component file metadata structures having a corresponding size indicated by the first vector.
18. A system, comprising:
a processor configured to:
determine that a size associated with a content file is greater than a threshold size;
store contents of the content file split across a plurality of component files, wherein a component file of the component files is configured to store a portion of the contents of the content file;
generate a plurality of component file metadata structures for the plurality of component files of the content file, wherein a component file metadata structure of the plurality of component file metadata structures corresponds to one of the plurality of component files of the content file, wherein each of the plurality of component file metadata structures includes a corresponding root node; and
update metadata, for the content file, to reference the plurality of component file metadata structures for the plurality of component files, wherein a node of the metadata is configured to track different sizes of portions of the content file stored in different component files of the plurality of component files, wherein file metadata of the content file is split across the plurality of component file metadata structures and each component file metadata structure of the plurality of component file metadata structures specifies a corresponding structure organizing data components for a corresponding portion of the content file; and
a memory coupled to the processor and configured to provide the processor with instructions.
19. The method of claim 1 , wherein the node of the metadata is configured to store information that indicates which component file metadata structure of the plurality of component file metadata structures is associated with which component file of the plurality of component files.