Maintaining I/O transaction metadata in log-with-index structure
An information handling system maintains a log-with-index (LWI) structure that includes a log for inserting and deleting records, each of which includes a key, and a binary index for maintaining an index of log records. The log may be stored to disk for persistence. Insertions of new records into the log may occur sequentially to ensure adequate performance. The index tree may be maintained in memory for performance. When the log is full, log records may be written in key-sorted order to a new tablet in a tablet library. Two tablets may be merged from time to time. Merging may include iteratively performing a plurality of atomic merges for each of a plurality of atomic portions. During atomic merges, tablet index data may be modified in copy-on-write fashion to preserve existing data until the atomic merge completes.
1. A storage system, comprising:
a storage controller, wherein the storage controller communicates a read/write operation to a dataset responsive to receiving an input/output (I/O) transaction from a host;
a metadata module, wherein the metadata module interfaces with the storage controller to communicate a metadata transaction to a metadata store including a log with index (LWI) resource including a transaction log and an index tree, wherein the metadata transaction indicates a data reference address and a data pattern address associated with the I/O transaction and wherein the metadata module is configured to perform operations comprising:
inserting a transaction record corresponding to the metadata transaction into the transaction log, wherein the transaction record includes a key, wherein the key is indicative of the data pattern address and the data reference address, and wherein an ordering in which the transaction records are stored in the transaction log indicates a chronological order in which corresponding metadata transactions were received;
updating the index tree to include an index to the transaction record wherein the index tree is indicative of a key based ordering of the transaction records stored in the transaction log;
clearing the LWI resource from time to time in response to a log full signal, wherein clearing the LWI resource includes:
creating a new transaction tablet in a tablet library, wherein the tablet library includes one or more older transaction tablets;
storing the transaction records to the new transaction tablet wherein records in the new transaction table are ordered in accordance with their corresponding keys; and
clearing the transaction log and the index tree of the transaction records; and
responsive to detecting a tablet merge signal, merging a first tablet in the tablet library with a second tablet in the tablet library to produce a merged tablet, wherein merging the first tablet and the second tablet comprises performing a plurality of atomic merges corresponding to a plurality of atomic portions of the first and second tablets, wherein each atomic merge includes merging an atomic portion of the first tablet with a corresponding atomic portion of the second tablet to form an atomic portion of the merged tablet.
2. The storage system of claim 1 , wherein each record is associated with a key-value pair comprising a key field indicating the key and a value field indicating a value associated with the key.
3. The storage system of claim 1 , wherein each record includes:
a presence bit for distinguishing between insertion transactions and deletion transactions; and
a sequence field for storing a current sequence value, wherein the current sequence value is assigned to each record in the transaction log; and
wherein the current sequence value is assigned to the new transaction tablet responsive to the clearing of the LWI resource and wherein the current sequence value is incremented each time the transaction log is cleared.
4. The storage system of claim 1 , wherein the index tree comprises a binary tree.
5. The storage system of claim 1 , wherein the transaction log comprises a nonvolatile storage structure and wherein the index tree comprises a volatile memory structure.
6. The storage system of claim 1 , wherein the metadata transaction includes one or more data deduplication transactions for identifying two or more references to a particular data pattern and assigning a single data pattern address to each of the two or more references.
7. The storage system of claim 1 , wherein the operations include maintaining a tablet index for accessing records stored in the tablet library; and
wherein the merging of an atomic portion of the first tablet with a corresponding atomic portion of the second tablet includes updating a portion of the tablet index corresponding to the atomic portion.
8. The storage system of claim 7 , wherein each transaction tablet in the tablet library is formatted into one or more fixed-size pages and wherein each atomic portion comprises a particular number of the fixed-size pages.
9. The storage system of claim 7 , wherein the updating of a portion of the tablet index includes copy-on-write updating a particular node of the tablet index, wherein the particular node includes original node data and wherein copy-on-write updating of the particular node includes creating a copy of the particular node, inserting modified data in the particular node, wherein the operations further include:
preserving the particular node and the copy of the particular node until successful completion of the atomic merge; and
responsive to completing the atomic merge, releasing the atomic portion of the first tablet, the atomic portion of the second tablet, and the copy of the particular node.
10. The storage system of claim 9 , wherein the tablet index includes:
root nodes, wherein the root nodes include a root node for the first tablet, a root node for the second tablet, and a root node for the merged table; and
a super root node comprising a parent node of the root nodes;
wherein said updating of the portion of the tablet index includes:
copy on write updating of the super root node and the root nodes during the atomic merge; and
responsive to successful completion of the atomic merge, committing the modified data in the tablet index and releasing the atomic portions of the first and second tablets.
11. The storage system of claim 1 , wherein the log full signal is asserted responsive to utilization of the transaction log exceeding a threshold selected from:
a percentage utilization threshold;
a record count threshold; and
a byte size threshold.
12. A data storage method, comprising:
detecting a metadata transaction sent to a metadata store, wherein the metadata store includes a log with index (LWI) resource, wherein the LWI resource includes a transaction log and an index tree, wherein the metadata transaction indicates a data reference address and a data pattern address associated with an I/O transaction;
inserting a transaction record corresponding to the metadata transaction into the transaction log, wherein the transaction record includes a key, wherein the key is indicative of the data pattern address and the data reference address, and wherein an ordering of in which the transaction records are stored in the transaction log indicates a chronological order in which corresponding metadata transactions were received;
updating the index tree to include an index to the transaction record wherein the index tree is indicative of a key based ordering of the transaction records stored in the transaction log;
clearing the LWI resource from time to time in response to a log full signal, wherein clearing the LWI resource includes:
creating a new transaction tablet in a tablet library, wherein the tablet library includes one or more older transaction tablets;
storing the transaction records to the new transaction tablet wherein records in the new transaction table are ordered in accordance with their corresponding keys; and
clearing the transaction log and the index tree of the transaction records; and
responsive to detecting a tablet merge signal, merging a first tablet in the tablet library with a second tablet in the tablet library to produce a merged tablet, wherein merging the first tablet and the second tablet comprises performing a plurality of atomic merges corresponding to a plurality of atomic portions of the first and second tablets, wherein each of atomic merge includes merging an atomic portion of the first tablet with a corresponding atomic portion of the second tablet to form an atomic portion of the merged tablet.
13. The method of claim 12 , wherein each record associated with an insertion transaction with a key-value pair comprising a key field indicating a key and a value field.
14. The method of claim 12 , wherein each record includes:
a presence bit for distinguishing between insertion transactions and deletion transactions; and
a sequence field storing a current sequence value of the LWI resource, wherein each record in the transaction log has the current sequence value; and
wherein the current sequence value of the LWI resource is incremented each time the transaction log is cleared.
15. The method of claim 12 , wherein the index tree comprises a binary tree.
16. The method of claim 12 , wherein the transaction log is maintained in nonvolatile storage and the index tree is maintained in memory.
17. The method of claim 12 , further comprising:
maintaining a tablet index for accessing records stored in the tablet library; and wherein the merging of an atomic portion of the first tablet with a corresponding atomic portion of the second tablet includes updating a portion of the tablet index corresponding to the atomic portion.
18. The method of claim 17 , wherein each atomic portion comprises a tablet portion corresponding to a particular range of the keys.
19. The method of claim 18 , wherein each atomic portion comprises a plurality of fixed-size pages and wherein atomic portions comprise a particular number of tablet pages.
20. The method of claim 19 , further comprising:
maintaining the tablet index as copy-on-write data wherein said updating of the tablet index preserves node data until the atomic merge is committed to the merged tablet and the atomic portions of the first and second tablets are released.