Methods and systems for efficient metadata management
Methods, computer program products, computer systems, and the like for efficient metadata management are disclosed, which can include receiving a subunit of storage, storing a first metadata portion of the subunit of storage in a first unit of storage, and storing a second metadata portion of the subunit of storage in a second unit of storage.
1. A method comprising:
determining a storage type of a storage unit for storage of a subunit of storage, wherein
the subunit of storage comprises metadata for data, and
the determining is based, at least in part, on a characteristic of the data;
in response to a determination that the subunit of storage should be stored in a unit of storage of a first storage type, storing the subunit of storage in a first unit of storage, wherein
the first unit of storage is the unit of storage of the first storage type,
the unit of storage of the first storage type comprises a first plurality of subunits of storage and is configured to be copied in its entirety, as part of a backup operation, and
permit access to one or more of the first plurality of subunits of storage individually, and
the unit of storage of the first storage type is configured to be copied in its entirety by virtue of the first plurality of subunits of storage being copied using a single copy operation performed on the unit of storage of the first storage type; and
in response to a determination that the subunit of storage may be stored in a unit of storage of a second storage type, storing the subunit of storage in a second unit of storage, wherein
the second unit of storage is either of the unit of storage of the first storage type or the unit of storage of the second storage type, and
the unit of storage of the second storage type comprises a second plurality of subunits of storage and is configured to permit copying of one or more of the second plurality of subunits of storage individually, as part of the backup operation.
2. The method of claim 1 , further comprising:
storing the subunit of storage in a third unit of storage, wherein
the subunit of storage comprises a first metadata portion, a second metadata portion, and a data portion,
the data portion comprises the data,
the first metadata portion is a header portion of the subunit of storage, and
the second metadata portion is an extent map portion of the subunit of storage.
3. The method of claim 2 , further comprising:
determining a size of the subunit of storage, wherein
the subunit of storage is received at a storage system,
the storage system comprises
the first unit of storage,
a third unit of storage, and
a fourth unit of storage,
the third unit of storage is of the first storage type, and
the fourth unit of storage is of the second storage type;
comparing the size of the subunit of storage and a size threshold;
in response to a result of the comparing indicating the subunit of storage should be stored in the unit of storage of the first storage type,
selecting the third unit of storage as the second unit of storage, and
performing the storing the second metadata portion of the subunit of storage in the second unit of storage; and
in response to the result of the comparing indicating the subunit of storage should be stored in the unit of storage of the second storage type,
selecting the fourth unit of storage as the second unit of storage, and
performing the storing the second metadata portion of the subunit of storage in the second unit of storage.
4. The method of claim 3 , further comprising:
determining an access frequency of the subunit of storage;
comparing the access frequency and an access frequency threshold; and
in response to the comparing the access frequency and the access frequency threshold indicating that the subunit of storage should be stored in the second unit of storage, storing the subunit of storage in the second unit of storage, without regard to the result of the comparing the size of the subunit of storage and the size threshold, rather than storing the subunit of storage in the first unit of storage.
5. The method of claim 3 , further comprising:
creating a partition;
creating the third unit of storage in the partition, wherein
the third unit of storage is a map volume, and
a volume type of the map volume is a filed-based loop volume; and
creating the fourth unit of storage in the partition, wherein
the fourth unit of storage is a local volume, and
a volume type of the local volume is a file-based volume.
6. The method of claim 5 , further comprising:
creating the first unit of storage in the partition, wherein
the first unit of storage is a header volume, and
a volume type of the header volume is the filed-based loop volume.
7. The method of claim 6 , wherein the subunit of storage is a file, and the method further comprises:
storing header information for the file in a header file in the header volume.
8. The method of claim 5 , further comprising:
creating another partition;
creating a fifth unit of storage in the another partition, wherein
the fifth unit of storage is another map volume, and
the another map volume is the filed-based loop volume; and
creating a sixth unit of storage in the another partition, wherein
the sixth unit of storage is another local volume, and
a volume type of the another local volume is the file-based volume.
9. The method of claim 3 , further comprising:
further in response to the result of the comparing the size of the subunit of storage and the size threshold indicating the subunit of storage should be stored in the second unit of storage,
determining whether an amount of storage space in the second unit of storage should be increased, and
in response to a determination that the amount of storage space in the second unit of storage should be increased, increasing the amount of storage space in the second unit of storage.
10. The method of claim 9 , wherein the determining comprises:
determining whether storage space remains in the second unit of storage sufficient to store the subunit of storage.
11. The method of claim 9 , wherein the amount of storage space in the second unit of storage is an amount of unused storage space, and the determining comprises:
determining whether the amount of unused storage space is below a threshold of available storage space for the second unit of storage.
12. The method of claim 1 , wherein
access to the one or more of the second plurality of subunits of storage comprises the copying the one or more of the second plurality of subunits of storage individually.
13. The method of claim 1 , wherein
the subunit of storage is a file that comprises the data and the metadata,
the unit of storage of the first storage type is a filed-based loop volume, and
the unit of storage of the second storage type is a file-based volume.
14. The method of claim 1 , wherein
the data is stored in a data file, and
the characteristic is a size of the data file and/or an access frequency of the data file.
15. The method of claim 1 , wherein
if the second unit of storage is unit of storage of the first storage type, storing the subunit of storage in the first unit of storage and storing the subunit of storage in the second unit of storage store the subunit of storage in the unit of storage of the first storage type.
16. A non-transitory computer-readable storage medium, comprising program instructions, which, when executed by one or more processors of a computing system, perform a method comprising:
determining a storage type of a storage unit for storage of a subunit of storage, wherein
the subunit of storage comprises metadata for data, and
the determining is based, at least in part, on a characteristic of the data;
in response to a determination that the subunit of storage should be stored in a unit of storage of a first storage type, storing the subunit of storage in a first unit of storage, wherein
the first unit of storage is the unit of storage of the first storage type,
the unit of storage of the first storage type comprises a first plurality of subunits of storage and is configured to
be copied in its entirety, as part of a backup operation, and
permit access to one or more of the first plurality of subunits of storage individually, and
the unit of storage of the first storage type is configured to be copied in its entirety by virtue of the first plurality of subunits of storage being copied using a single copy operation performed on the unit of storage of the first storage type; and
in response to a determination that the subunit of storage may be stored in a unit of storage of a second storage type, storing the subunit of storage in a second unit of storage, wherein
the second unit of storage is either of the unit of storage of the first storage type or the unit of storage of the second storage type, and
the unit of storage of the second storage type comprises a second plurality of subunits of storage and is configured to permit copying of one or more of the second plurality of subunits of storage individually, as part of the backup operation.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises:
determining a size of the subunit of storage, wherein
the subunit of storage comprises a first metadata portion, a second metadata portion, and a data portion,
the data portion comprises the data,
the subunit of storage is received at a storage system,
the storage system comprises
the first unit of storage,
a third unit of storage, and
a fourth unit of storage,
the third unit of storage is of the first storage type, and
the fourth unit of storage is of the second storage type;
comparing the size of the subunit of storage and a size threshold;
in response to a result of the comparing indicating the subunit of storage should be stored in the unit of storage of the first storage type,
selecting the third unit of storage as the second unit of storage, and
performing the storing the second metadata portion of the subunit of storage in the second unit of storage; and
in response to the result of the comparing indicating the subunit of storage should be stored in the unit of storage of the second storage type,
selecting the fourth unit of storage as the second unit of storage, and
performing the storing the second metadata portion of the subunit of storage in the second unit of storage.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the method further comprises:
determining an access frequency of the subunit of storage;
comparing the access frequency and an access frequency threshold; and
in response to the comparing the access frequency and the access frequency threshold indicating that the subunit of storage should be stored in the second unit of storage, storing the subunit of storage in the second unit of storage, without regard to the result of the comparing the size of the subunit of storage and the size threshold, rather than storing the subunit of storage in the first unit of storage.
19. The non-transitory computer-readable storage medium of claim 17 , wherein the method further comprises:
creating a partition;
creating the first unit of storage in the partition, wherein
the first unit of storage is a header volume, and
a volume type of the header volume is a filed-based loop volume;
creating the third unit of storage in the partition, wherein
the third unit of storage is a map volume, and
a volume type of the map volume is the filed-based loop volume; and
creating the fourth unit of storage in the partition, wherein
the fourth unit of storage is a local volume, and
a volume type of the local volume is a file-based volume.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the method further comprises:
creating another partition;
creating a fifth unit of storage in the another partition, wherein
the fifth unit of storage is another map volume, and
the another map volume is the filed-based loop volume; and
creating a sixth unit of storage in the another partition, wherein
the sixth unit of storage is another local volume, and
a volume type of the another local volume is the file-based volume.
21. The non-transitory computer-readable storage medium of claim 17 , wherein the method further comprises:
further in response to the result of the comparing the size of the subunit of storage and the size threshold indicating the subunit of storage should be stored in the second unit of storage,
determining whether an amount of storage space in the second unit of storage should be increased, wherein
the amount of storage space in the second unit of storage is an amount of unused storage space, and
the determining comprises
determining whether storage space remains in the second unit of storage sufficient to store the subunit of storage, or
determining whether the amount of unused storage space is below a threshold of available storage space for the second unit of storage, and
in response to a determination that the amount of storage space in the second unit of storage should be increased, increasing the amount of storage space in the second unit of storage.
22. A computing system comprising:
one or more processors; and
a computer-readable storage medium coupled to the one or more processors, comprising program instructions, which, when executed by the one or more processors, perform a method comprising
determining a storage type of a storage unit for storage of a subunit of storage,
wherein
the subunit of storage comprises metadata for data, and
the determining is based, at least in part, on a characteristic of the data,
in response to a determination that the subunit of storage should be stored in a unit of storage of a first storage type, storing the subunit of storage in a first unit of storage, wherein
the first unit of storage is the unit of storage of the first storage type,
the unit of storage of the first storage type comprises a first plurality of subunits of storage and is configured to
be copied in its entirety, as part of a backup operation, and
permit access to one or more of the first plurality of subunits of storage individually, and
the unit of storage of the first storage type is configured to be copied in its entirety by virtue of the first plurality of subunits of storage being copied using a single copy operation performed on the unit of storage of the first storage type, and
in response to a determination that the subunit of storage may be stored in a unit of storage of a second storage type, storing the subunit of storage in a second unit of storage, wherein
the second unit of storage is either of the unit of storage of the first storage type or the unit of storage of the second storage type, and
the unit of storage of the second storage type comprises a second plurality of subunits of storage and is configured to permit copying of one or more of the second plurality of subunits of storage individually, as part of the backup operation.