Space accounting for data storage usage
An aspect of space accounting includes providing a counter for each of a plurality of key/value pairs for storage system operations. The key uniquely identifies a logical storage unit on which operations associated with the key are performed, the value identifies an amount of data usage with respect to the operations performed on the logical storage unit, and the counter tracks a type of the operations. An aspect also includes tracking, via corresponding counters, updates to a read/write (RW) tablet associated with a Cuckoo tree. The tracking includes updating a filter to include instances of key/value pairs and the RW tablet in which the pairs are stored. An aspect further includes tracking a plurality of read-only (RO) tablets associated with the Cuckoo tree. The tracking includes updating the filter to include instances of key/value pairs and one or more of the RO tablets in which the pairs are stored.
1. A method of space accounting for data storage usage, the method comprising:
providing a counter for each of a plurality of key/value-operation type pairs corresponding to metadata for operations performed at a storage system, the key uniquely identifying a logical storage unit on which operations associated with the key are performed, the value-operation identifying an amount of data usage with respect to the operations performed on the logical storage unit, and a type of the operations;
tracking, via corresponding counters, updates to a plurality of read/write (RW) tablets within a Cuckoo tree, the tracking including updating a filter to include instances of key/value-operation type pairs and the RW tablets in which the pairs are stored;
tracking updates to a plurality of read-only (RO) tablets in the Cuckoo tree, the tracking including updating the filter to include instances of key/value-operation type pairs and one or more of the RO tablets in which the pairs are stored;
monitoring a number of keys added to each RW tablet of the plurality of RW tablets, and upon determining the number of added keys to a RW tablet of the plurality of RW tablets has reached a threshold value: converting the RW tablet to an RO tablet, creating a new RW tablet, and adding the new RW tablet to the Cuckoo tree;
monitoring a number of RO tablets that are present in the Cuckoo tree, and upon determining the number of RO tablets has reached a threshold count: merging the number of RO tablets and storing merged data from the number of RO tablets into a consolidated tablet; and
upon receiving a query for a key, searching the consolidated tablet for the key.
2. The method of claim 1 , wherein the RO tablet is positioned ahead of the plurality of RO tablets in the Cuckoo tree.
3. The method of claim 1 , further comprising:
monitoring a number of merge operations performed on the RO tablets over time, the merge operations producing a plurality of consolidated tablets; and
upon determining a threshold value has been reached for the merge operations, merging the plurality of consolidated tablets into a single, consolidated tablet.
4. The method of claim 1 , wherein the operation type is one of compression and non-compression, wherein factoring in the operation type includes factoring in compressed data resulting from a corresponding operation in the amount of data usage.
5. The method of claim 1 , wherein the operation type is one of an allocated write and an overwrite, wherein factoring in the operation type includes factoring in how much of the data usage correlates to allocated writes and how much of the data usage correlates to overwrites in determining a collective amount of data usage.
6. The method of claim 1 , further comprising:
providing a hash data structure for the Cuckoo tree, the hash data structure including a hash function performed on each of the keys, wherein hashed keys are stored in the hash data structure;
wherein searching the filter for the key includes searching the hash data structure for the key prior to searching the filter.
7. A system for implementing space accounting for data storage usage, the system includes:
a memory comprising computer-executable instructions; and
a processor executing the computer-executable instructions, the computer-executable instructions when executed by the processor cause the processor to perform operations comprising:
providing a counter for each of a plurality of key/value-operation type pairs corresponding to metadata for operations performed at a storage system, the key uniquely identifying a logical storage unit on which operations associated with the key are performed, the value-operation identifying an amount of data usage with respect to the operations performed on the logical storage unit, and a type of the operations;
tracking, via corresponding counters, updates to a read/write (RW) tablet within a Cuckoo tree, the tracking including updating a filter to include instances of key/value-operation type pairs and the RW tablet in which the pairs are stored;
tracking updates to a plurality of read-only (RO) tablets in the Cuckoo tree, the tracking including updating the filter to include instances of key/value-operation type pairs and one or more of the RO tablets in which the pairs are stored;
monitoring a number of keys added to each RW tablet of the plurality of RW tablets, and upon determining the number of added keys to a RW tablet of the plurality of RW tablets has reached a threshold value: converting the RW tablet to an RO tablet, creating a new RW tablet, and adding the new RW tablet to the Cuckoo tree;
monitoring a number of RO tablets that are present in the Cuckoo tree, and upon determining the number of RO tablets has reached a threshold count: merging the number of RO tablets and storing merged data from the number of RO tablets into a consolidated tablet; and
upon receiving a query for a key, searching the consolidated tablet for the key.
8. The system of claim 7 , wherein the RO tablet is positioned ahead of the plurality of RO tablets in the Cuckoo tree.
9. The system of claim 7 , wherein the operations further comprise:
monitoring a number of merge operations performed on the RO tablets over time, the merge operations producing a plurality of consolidated tablets; and
upon determining a threshold value has been reached for the merge operations, merging the plurality of consolidated tablets into a single, consolidated tablet.
10. The system of claim 7 , wherein the operation type is one of compression and non-compression, wherein factoring in the operation type includes factoring in compressed data resulting from a corresponding operation in the amount of data usage.
11. The system of claim 7 , wherein the operation type is one of an allocated write and an overwrite, wherein factoring in the operation type includes factoring in how much of the data usage correlates to allocated writes and how much of the data usage correlates to overwrites in determining a collective amount of data usage.
12. The system of claim 7 , wherein the operations further comprise:
providing a hash data structure for the Cuckoo tree, the hash data structure including a hash function performed on each of the keys, wherein hashed keys are stored in the hash data structure;
wherein searching the filter for the key includes searching the hash data structure for the key prior to searching the filter.
13. A computer program product for implementing space accounting for data storage usage, the computer program product embodied on a non-transitory computer readable medium, the computer program product including instructions that, when executed by a computer, causes the computer to perform operations comprising:
providing a counter for each of a plurality of key/value-operation type pairs corresponding to metadata for operations performed at a storage system, the key uniquely identifying a logical storage unit on which operations associated with the key are performed, the value-operation identifying an amount of data usage with respect to the operations performed on the logical storage unit, and a type of the operations;
tracking, via corresponding counters, updates to a read/write (RW) tablet within a Cuckoo tree, the tracking including updating a filter to include instances of key/value-operation type pairs and the RW tablet in which the pairs are stored;
tracking updates to a plurality of read-only (RO) tablets in the Cuckoo tree, the tracking including updating the filter to include instances of key/value-operation type pairs and one or more of the RO tablets in which the pairs are stored;
monitoring a number of keys added to each RW tablet of the plurality of RW tablets, and upon determining the number of added keys to a RW tablet of the plurality of RW tablets has reached a threshold value: converting the RW tablet to an RO tablet, creating a new RW tablet, and adding the new RW tablet to the Cuckoo tree;
monitoring a number of RO tablets that are present in the Cuckoo tree, and upon determining the number of RO tablets has reached a threshold count: merging the number of RO tablets and storing merged data from the number of RO tablets into a consolidated tablet; and
upon receiving a query for a key, searching the consolidated tablet for the key.
14. The computer program product of claim 13 , wherein the operations further comprise:
upon receiving a query for a key:
searching the filter for the key;
receiving at least one response from the filter indicating one or more of the RW tablet and the RO tablets in which the key is found;
searching, based on the at least one response, only those of the one or more of the RW tablets and the RO tablets in which the key is present; and
upon identifying entries for the key in the one or more of the RW tablets and the RO tablets, retrieving corresponding values in the entries, summing values in the entries, and returning a summed value to a source of the query, the summed value indicating a collective amount of data usage associated with the key factoring in the operation type.