Large content file optimization
A size associated with a content file is determined to be greater than a threshold size. Contents of the content file split across a plurality of component files are stored. Metadata, for the content file, is updated to reference a plurality of component file metadata structures for the component files. A node of the metadata is configured to track different sizes of portions of the content file stored in different component files of the plurality of component files. File metadata of the content file is split across the plurality of component file metadata structures and each component file metadata structure of the plurality of component file metadata structures specifies a corresponding structure organizing data components for a corresponding portion of the content file.
1. A method, comprising:
performing a backup of a primary system that includes a plurality of portions of a content file that has a size that is greater than a threshold size;
storing the plurality of portions of the content file; and
generating a tree data structure that provides a view of the primary system, wherein generating the tree data structure includes generating a plurality of component file metadata structures for each of the plurality of portions of the content file, wherein a component file metadata structure of the plurality of component file metadata structures corresponds to one of the portions of the content file, wherein each of the plurality of component file metadata structures includes a corresponding root node, wherein each of the plurality of component file metadata structures includes metadata that enables data chunks associated with a corresponding portion of the content file to be located, wherein the tree data structure includes a plurality of leaf nodes, wherein a first leaf node of the plurality of leaf nodes stores a first vector that indicates a size of corresponding content file data that is associated with a corresponding component file metadata structure.
2. The method of claim 1 , wherein at least two of the plurality of portions of the content file have a same size.
3. The method of claim 1 , wherein at least two of the plurality of portions of the content file have a different size.
4. The method of claim 1 , further comprising determining that the size of the content file is greater than the threshold size.
5. The method of claim 1 , further comprising determining that the size of the content file is greater than the threshold size after performing the backup.
6. The method of claim 1 , wherein the component file metadata structure includes a second plurality of leaf nodes.
7. The method of claim 6 , wherein each of the second plurality of leaf nodes is associated with a corresponding data brick, wherein the corresponding data brick is an identifier for one or more data chunks.
8. The method of claim 7 , wherein a last data brick of the plurality of leaf nodes has a particular capacity, wherein the last data brick is brick aligned in the event the last data brick is associated with one or more data chunks having the particular capacity.
9. The method of claim 7 , wherein a last data brick of the plurality of leaf nodes has a particular capacity, wherein in the event the last data brick of the plurality of leaf nodes is not brick aligned, an unused portion of the last data brick is reserved for the content file.
10. The method of claim 1 , wherein the first leaf node of the plurality of leaf nodes stores information that indicates which component file metadata structure of the plurality of component file metadata structures is associated with which portion of the content file.
11. The method of claim 1 , wherein a plurality of sequential component file metadata structures associated with the content file have a same corresponding size.
12. The method of claim 11 , wherein the first vector utilizes run length encoding for the plurality of sequential component file metadata structures associated with the content file that have the same corresponding size.
13. The method of claim 12 , wherein the first leaf node of the plurality of leaf nodes stores a second vector that indicates a number of the sequential component file metadata structures that have the same corresponding size.
14. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
performing a backup of a primary system that includes a plurality of portions of a content file that has a size that is greater than a threshold size;
storing plurality of portions of content file; and
generating a tree data structure that provides a view of the primary system, wherein generating the tree data structure includes generating a plurality of component file metadata structures for each of the plurality of portions of the content file, wherein a component file metadata structure of the plurality of component file metadata structures corresponds to one of the portions of the content file, wherein each of the plurality of component file metadata structures includes a corresponding root node, wherein each of the plurality of component file metadata structures includes metadata that enables data chunks associated with a corresponding portion of the content file to be located, wherein the tree data structure includes a plurality of leaf nodes, wherein a first leaf node of the plurality of leaf nodes stores a first vector that indicates a size of corresponding content file data that is associated with a corresponding component file metadata structure.
15. The computer program product of claim 14 , further comprising determining that the size of the content file is greater than the threshold size.
16. The computer program product of claim 14 , further comprising determining that the size of the content file is greater than the threshold size after performing the backup.
17. The computer program product of claim 14 , wherein a plurality of sequential component file metadata structures associated with the content file have a same corresponding size.
18. The computer program product of claim 17 , wherein the first vector utilizes run length encoding for the plurality of sequential component file metadata structures associated with the content file that have the same corresponding size.
19. The computer program product of claim 18 , wherein the first leaf node of the plurality of leaf nodes stores a second vector that indicates a number of the sequential component file metadata structures that have the same corresponding size.
20. A system, comprising:
a processor configured to:
perform a backup of a primary system that includes a plurality of portions of a content file that has a size that is greater than a threshold size;
store the plurality of portions of the content file; and
generate a tree data structure that provides a view of the primary system, wherein to generate the tree data structure, the processor is configured to generate a plurality of component file metadata structures for each of the plurality of portions of the content file, wherein a component file metadata structure of the plurality of component file metadata structures corresponds to one of the portions of the content file, wherein each of the plurality of component file metadata structures includes a corresponding root node, wherein each of the plurality of component file metadata structures includes metadata that enables data chunks associated with a corresponding portion of the content file to be located, wherein the tree data structure includes a plurality of leaf nodes, wherein a first leaf node of the plurality of leaf nodes stores a first vector that indicates a size of corresponding content file data that is associated with a corresponding component file metadata structure; and
a memory coupled to the processor and configured to provide the processor with instructions.