Metadata compaction in a distributed storage system
Systems and methods for metadata compaction in a distributed storage system with a file system interface are described. A file system interface and an object storage system interface use a metadata index for mapping object identifiers from the object storage system to location identifiers for the file system. When the metadata index includes a number of entries for continuous data blocks with overlapping intervals, a defragmentation operation may generate a defragmented entry for a defragmentation interval overlapping the overlapping data blocks.
1. A system, comprising:
a file system interface configured to process file data requests for data from a file system client;
an object storage system interface configured to process object storage requests to an object storage system; and
a metadata index manager configured to:
manage a metadata index, wherein:
the metadata index includes a plurality of entries;
each entry of the plurality of entries includes:
an object identifier for the object storage system; and
a file system location identifier for a block of data; and
the plurality of entries includes a plurality of continuous data blocks; and
generate, responsive to a defragmentation operation, a defragmented entry in the metadata index, wherein the defragmented entry includes a defragmentation interval traversing the plurality of continuous data blocks.
2. The system of claim 1 , further comprising:
a compactor configured to:
determine the defragmentation interval for a first determined quantity of the plurality of entries;
read blocks of data for the defragmentation interval from the object storage system; and
write a defragmented block of data for the defragmentation interval in the object storage system, wherein writing the block of data for the defragmentation interval generates the defragmented entry in the metadata index.
3. The system of claim 2 , wherein:
the metadata index manager is further configured to:
add the plurality of entries in sequential order in the metadata index based on a timestamp associated with an operation associated with each entry;
access entries in the metadata index in reverse chronological order for new data operations; and
add the defragmented entry following the plurality of entries for the plurality of continuous data blocks.
4. The system of claim 2 , wherein:
the file system location identifier for each entry of the plurality of entries defines:
a block start location;
a block end location; and
a block length for an interval of the block of data; and
a new file system location identifier for the defragmented entry:
identifies the defragmented block of data spanning the defragmentation interval;
defines a defragmented block start location equal to an earliest block start location among the plurality of continuous data blocks; and
defines a defragmented block end location equal to a latest block end location among the plurality of continuous data blocks.
5. The system of claim 2 , wherein:
the compactor is further configured to:
remove entries in the plurality of entries preceding the defragmented entry that include a block of data within the defragmentation interval.
6. The system of claim 2 , further comprising:
a defragmentation initiator configured to:
determine an operation type for a storage operation; and
initiate, responsive to the operation type being a write operation, the defragmentation operation, wherein the compactor includes a write block for the write operation in the defragmentation interval.
7. The system of claim 2 , further comprising:
a defragmentation initiator configured to:
determine an operation type for a storage operation; and
initiate, responsive to the operation type being a read operation, the defragmentation operation, wherein the compactor includes a read interval for the read operation in the defragmentation interval.
8. The system of claim 7 , wherein the read interval is the defragmentation interval.
9. The system of claim 1 , further comprising:
a defragmentation initiator configured to:
determine a fragmentation value for the metadata index based on a relationship between:
a file size value for the file system client; and
an object size value for the object storage system;
determine a fragmentation threshold based on a performance impact value for the object storage system; and
initiate, responsive to the fragmentation value meeting the fragmentation threshold, the defragmentation operation.
10. The system of claim 1 , further comprising:
a defragmentation initiator configured to:
receive a maintenance operation request for the defragmentation operation; and
initiate, responsive to receiving the maintenance operation request, the defragmentation operation.
11. A computer-implemented method, comprising:
processing, through a file system interface, file data requests for data from a file system client;
processing, through an object storage system interface, object storage requests to an object storage system;
managing a metadata index, wherein:
the metadata index includes a plurality of entries;
each entry of the plurality of entries includes:
an object identifier for the object storage system; and
a file system location identifier for a block of data; and
the plurality of entries includes a plurality of continuous data blocks with continuous intervals; and
generating, responsive to a defragmentation operation, a defragmented entry in the metadata index, wherein the defragmented entry includes a defragmentation interval traversing the plurality of continuous data blocks.
12. The computer-implemented method of claim 11 , further comprising:
determining the defragmentation interval for a first determined quantity of the plurality of entries;
reading blocks of data for the defragmentation interval from the object storage system; and
writing a defragmented block of data for the defragmentation interval in the object storage system, wherein writing the defragmented block of data for the defragmentation interval generates the defragmented entry in the metadata index.
13. The computer-implemented method of claim 12 , wherein the managing the metadata index comprises:
adding the plurality of entries in sequential order in the metadata index based on a timestamp associated with an operation associated with each entry;
accessing entries in the metadata index in reverse chronological order for new data operations; and
adding the defragmented entry following the plurality of entries for the plurality of continuous data blocks.
14. The computer-implemented method of claim 12 , wherein:
the file system location identifier for each entry of the plurality of entries defines:
a block start location;
a block end location; and
a block length for an interval of the block of data; and
a new file system location identifier for the defragmented entry:
identifies the defragmented block of data spanning the defragmentation interval;
defines a defragmented block start location equal to an earliest block start location among the plurality of continuous data blocks; and
defines a defragmented block end location equal to a latest block end location among the plurality of continuous data blocks.
15. The computer-implemented method of claim 12 , further comprising:
removing entries in the plurality of entries preceding the defragmented entry that include a block of data within the defragmentation interval.
16. The computer-implemented method of claim 12 , further comprising:
determining an operation type for a storage operation; and
initiating, responsive to the operation type being a write operation, the defragmentation operation, wherein a write block for the write operation is in the defragmentation interval.
17. The computer-implemented method of claim 12 , further comprising:
determining an operation type for a storage operation; and
initiating, responsive to the operation type being a read operation, the defragmentation operation, wherein a read interval for the read operation is in the defragmentation interval.
18. The computer-implemented method of claim 11 , further comprising:
determining a fragmentation value for the metadata index based on a relationship between:
a file size value for the file system client; and
an object size value for the object storage system;
determining a fragmentation threshold based on a performance impact value for the object storage system; and
initiating, responsive to the fragmentation value meeting the fragmentation threshold, the defragmentation operation.
19. The computer-implemented method of claim 11 , further comprising:
receiving a maintenance operation request for the defragmentation operation; and
initiating, responsive to receiving the maintenance operation request, the defragmentation operation.
20. A system, comprising:
means for processing, through a file system interface, file data requests for data from a file system client;
means for processing, through an object storage system interface, object storage requests to an object storage system;
means for managing a metadata index, wherein:
the metadata index includes a plurality of entries;
each entry of the plurality of entries includes:
an object identifier for the object storage system; and
a file system interface location identifier for a block of data; and
the plurality of entries includes a plurality of continuous data blocks with continuous intervals; and
means for generating, responsive to a defragmentation operation, a defragmented entry in the metadata index, wherein the defragmented entry includes a defragmentation interval traversing the plurality of continuous data blocks.