Methods and systems for the movement of metadata between different storage types
Methods, computer program products, computer systems, and the like for efficient metadata management are disclosed, which can include determining whether a change in a status of data has occurred. In response to a determination that the change has occurred, such methods, computer program products, and computer systems can include determining whether a move condition has been met, and, in response to a determination that the move condition has been met, moving the metadata from the first storage unit to a second storage unit.
1. A method comprising:
determining whether a change in a status of data has occurred, wherein
the data is stored in a storage system that comprises a plurality of storage units,
the status is based, at least in part, on a characteristic of the data,
metadata for the data is stored in a first storage unit of the plurality of storage units,
the first storage unit is of one storage type,
the one storage type is one of a first storage type of a plurality of storage types or a second storage type of the plurality of storage types,
a storage unit of the first storage type comprises a first plurality of subunits of storage,
the storage unit of the first storage type is configured to be copied in its entirety, as part of a backup operation,
a unit of storage of the second storage type comprises a second plurality of subunits of storage, and
the unit of storage of the second storage type is configured to be copied in its entirety, as part of the backup operation, and to permit copying of one or more of the second plurality of subunits of storage individually, as part of the backup operation; and
in response to a determination that the change has occurred,
determining whether a move condition has been met, and
in response to a determination that the move condition has been met, moving the metadata from the first storage unit to a second storage unit, wherein
the second storage unit is of another storage type,
the another storage type is another of the first storage type or the second storage type, and
the another storage type is one of the plurality of storage types other than the one storage type.
2. The method of claim 1 , wherein the determining whether the move condition has been met comprises:
determining whether the characteristic matches the move condition.
3. The method of claim 2 , wherein the determining whether the characteristic matches the move condition comprises:
determining whether the characteristic meets or exceeds one or more criteria.
4. The method of claim 2 , wherein the determining whether the characteristic matches the move condition comprises:
for each criteria of a plurality of criteria,
determining whether a corresponding characteristic of a plurality of
corresponding characteristics meets or exceeds the each criteria, wherein the plurality of corresponding characteristics comprises the characteristic; and
in response to each of the plurality of corresponding characteristics matching a corresponding one of the plurality of criteria, performing the moving.
5. The method of claim 1 , wherein
the change in the status is a result of a data status event, and
the data status event is at least one of
a change in a frequency with which the data is accessed, or
a change in size of the data.
6. The method of claim 5 , wherein
the data comprises a file, and
the characteristic is an attribute of the file.
7. The method of claim 6 , wherein
the change is a result of at least one of
a file creation operation,
a file write operation, or
a file deletion operation.
8. The method of claim 1 , wherein the data comprises a file, and the method further comprises:
retrieving the status from a file status cache.
9. The method of claim 8 , further comprising:
updating the status in the file status cache upon the change, wherein
the change is a result of one or more operations comprising at least one of
a file creation operation,
a file write operation, or
a file deletion operation, and
the updating is performed by a daemon configured to scan the file status cache for one or more results of the one or more operations.
10. The method of claim 1 , wherein
the first storage type is a map volume, and
the second storage type is a local volume.
11. The method of claim 10 , wherein
the data is stored in a data volume, and
the data volume, the local volume, and the map volume are stored in a partition.
12. The method of claim 10 , wherein
the local volume is a loop volume,
the status is stored in a file status cache, and
the file status cache is stored in a header volume.
13. The method of claim 1 , further comprising:
performing a snapshot operation, wherein
the snapshot operation comprises
performing a volume copy operation on one or more storage units of the first storage type, and
performing a file copy operation on one or more storage units of the second storage type.
14. The method of claim 1 , further comprising:
determining a size of a subunit of storage, wherein
the subunit of storage is to be stored as part of the data,
the subunit of storage is to be stored as part of the data, the subunit of storage is received at a storage system, and
the storage system comprises
the first storage unit, and
the second storage unit;
comparing the size of the subunit of storage and a size threshold;
in response to a result of the comparing indicating metadata associated with the subunit of storage should be stored in a storage unit of the first storage type,
selecting the unit of storage of the first storage type, and
performing the moving from a storage unit of the first storage type to a storage unit of the second storage type; and
in response to the result of the comparing indicating the metadata associated with the subunit of storage should be stored in a storage unit of the second storage type,
selecting the unit of storage of the second storage type, and
performing the moving from a storage unit of the second storage type to a storage unit of the first storage type.
15. The method of claim 14 , further comprising:
determining an access frequency of another subunit of storage, wherein
the another subunit of storage is comprised in the data;
comparing the access frequency and an access frequency threshold; and
in response to the comparing the access frequency and the access frequency threshold indicating that the moving should be performed, performing the moving.
16. A non-transitory computer-readable storage medium, comprising program instructions, which, when executed by one or more processors of a computing system, perform a method comprising:
determining whether a change in a status of data has occurred, wherein
the data is stored in a storage system that comprises a plurality of storage units,
the status is based, at least in part, on a characteristic of the data,
metadata for the data is stored in a first storage unit of the plurality of storage units,
the first storage unit is of one storage type,
the one storage type is one of a first storage type of a plurality of storage types or a second storage type of the plurality of storage types,
a storage unit of the first storage type comprises a first plurality of subunits of storage,
the storage unit of the first storage type is configured to be copied in its entirety, as part of a backup operation,
a unit of storage of the second storage type comprises a second plurality of subunits of storage, and
the unit of storage of the second storage type is configured to be copied in its entirety, as part of the backup operation, and to permit copying of one or more of the second plurality of subunits of storage individually, as part of the backup operation; and
in response to a determination that the change has occurred,
determining whether a move condition has been met, and
in response to a determination that the move condition has been met, moving the metadata from the first storage unit to a second storage unit, wherein
the second storage unit is of another storage type,
the another storage type is another of the first storage type or the second storage type, and
the another storage type is one of the plurality of storage types other than the one storage type.
17. The non-transitory computer-readable storage medium of claim 16 , wherein
the change in the status is a result of a data status event, and
the data status event is at least one of
a change in a frequency with which the data is accessed, or
a change in size of the data.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the data comprises a file, and the method further comprises:
retrieving the status from a file status cache; and
updating the status in the file status cache upon the change, wherein
the change is a result of one or more operations comprising at least one of
a file creation operation,
a file write operation, or
a file deletion operation, and
the updating is performed by a daemon configured to scan the file status cache for one or more results of the one or more operations.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises:
performing a snapshot operation, wherein
the snapshot operation comprises
performing a volume copy operation on one or more storage units of the first storage type, and
performing a file copy operation on one or more storage units of the second storage type.
20. A computing system comprising:
one or more processors; and
a non-transitory computer-readable storage medium coupled to the one or more
processors, comprising program instructions, which, when executed by the one or more processors, perform a method comprising
determining whether a change in a status of data has occurred, wherein
the data is stored in a storage system that comprises a plurality of storage units,
the status is based, at least in part, on a characteristic of the data,
metadata for the data is stored in a first storage unit of the plurality of storage units,
the first storage unit is of one storage type,
the one storage type is one of a first storage type of a plurality of storage types or a second storage type of the plurality of storage types,
a storage unit of the first storage type comprises a first plurality of subunits of storage,
the storage unit of the first storage type is configured to be copied in its entirety, as part of a backup operation,
a unit of storage of the second storage type comprises a second plurality of subunits of storage, and
the unit of storage of the second storage type is configured to be copied in its entirety, as part of the backup operation, and to permit copying of one or more of the second plurality of subunits of storage individually, as part of the backup operation; and
in response to a determination that the change has occurred,
determining whether a move condition has been met, and
in response to a determination that the move condition has been met, moving the metadata from the first storage unit to a second storage unit, wherein
the second storage unit is of another storage type,
the another storage type is another of the first storage type or the second storage type, and
the another storage type is one of the plurality of storage types other than the one storage type.