Systems and methods for compressing keys in hierarchical data structures
The present disclosure describes systems and methods for enhancing the space utilization of storing keys in hierarchical data structures, such as B+Tree leaf blocks, by partially or fully compressing the keys. Partially compressing the keys may include storing certain keys as offsets from a start key. Fully compressing the keys may include setting a “type” value included in the header of the leaf block to a value indicating that the keys are implicitly represented by an ordinal key index, which defines the ordinal arrangement of keys in the leaf block. Partially or fully compressing keys, according to the implementations described herein, reduces the amount of space needed to store keys in a leaf block of a B+Tree.
1. A file system, comprising:
one or more processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more processors, cause the one or more processors to perform operations including:
accessing a set of keys, each key of the set of keys being associated with a pointer to one or more data blocks of a block-oriented storage device, the set of keys being arranged according to an ordinal key index, and the set of keys including a start key being first from amongst the set of keys;
defining a hierarchical data structure configured to store the set of keys, the hierarchical data structure including a leaf block, the leaf block being configured to store a first subset of keys of the set of keys according to the ordinal key index;
storing first one or more keys of the first subset of keys in the leaf block, the first one or more keys including at least the start key;
determining a set of offsets, each offset of the set of offsets corresponding to a key of the first subset of keys;
determining whether the first subset of keys is contiguous;
in response to determining that the first subset of keys is contiguous, compressing the first subset of keys by storing one or more first offsets of the set of offsets in the leaf block, the one or more first offsets corresponding to second one or more keys of the first subset of keys; and
in response to determining that the subset of the set of keys are not contiguous, compressing the first subset of keys by storing one or more second offsets of the set of offsets in the leaf block in a new data structure including a set of records, the one or more second offsets being contiguous in the new data structure, and each record of the set of records of the new data structure including a key after the start key of the first subset of the set of keys and the associated offset of the key.
2. The file system of claim 1 , wherein determining the set of offsets further comprises comparing the start key with each other key of the first subset of keys, wherein each offset of the set of offsets corresponds to a difference between the start key and a subsequent key of the first subset of keys.
3. The file system of claim 2 , wherein each particular offset of the one or more offsets are stored:
as a partial keys corresponding to the difference between the start key and a subsequent key of the first subset of keys in the leaf block subsequent to the start key, according to the ordinal key index.
4. The file system of claim 1 , wherein the operation of compressing the first subset of the set of keys further comprises:
determining that each offset of the set of offsets has a same size; and
in response to determining that each offset of the set of offsets has a same size, storing one offset of the one or more offsets in a header of the leaf block, the one offset of the one or more offsets indicating the offset for each key of the first subset of keys is the same size and is based on the ordinal key index.
5. The file system of claim 4 , wherein the ordinal key index and the one offset of the one or more offsets implicitly represent each key of the third subset of keys.
6. The file system of claim 1 , wherein the hierarchical data structure is a B+Tree, wherein each key of the subset of the set of keys represents a file offset of a file, and wherein the B+Tree is used to store a file layout tree representing the file.
7. A computer-implemented method comprising:
accessing a set of keys, each key of the set of keys being associated with a pointer to one or more data blocks of a block-oriented storage device, the set of keys being arranged according to an ordinal key index, and the set of keys including a start key being first from amongst the set of keys;
defining a hierarchical data structure configured to store the set of keys, the hierarchical data structure including a leaf block, the leaf block being configured to store a first subset of keys of the set of keys according to the ordinal key index;
storing first one or more keys of the first subset of keys in the leaf block, the first one or more keys including at least the start key;
determining a set of offsets, each offset of the set of offsets corresponding to a key of the first subset of keys;
determining whether the first subset of keys is contiguous;
in response to determining that the first subset of keys is contiguous, compressing the first subset of keys by
storing one or more first offsets of the set of offsets in the leaf block, the one or more first offsets corresponding to second one or more keys of the first subset of keys; and
in response to determining that the subset of the set of keys are not contiguous, compressing the first subset of keys by storing one or more second offsets of the set of offsets in the leaf block in a new data structure including a set of records, the one or more second offsets being contiguous in the new data structure, and each record of the set of records of the new data structure including a key after the start key of the first subset of the set of keys and the associated offset of the key.
8. The computer-implemented method of claim 7 , wherein determining the set of offsets further comprises comparing the start key with each other key of the first subset of keys, wherein each offset of the set of offsets corresponds to a difference between the start key and a subsequent key of the first subset of keys.
9. The computer-implemented method of claim 8 , wherein each particular offset of the one or more offsets are stored:
as a partial keys corresponding to the difference between the start key and a subsequent key of the first subset of keys in the leaf block subsequent to the start key, according to the ordinal key index.
10. The computer-implemented method of claim 7 , wherein the operation of compressing the first subset of the set of keys further comprises:
determining that each offset of the set of offsets has a same size; and
in response to determining that each offset of the set of offsets has a same size, storing one offset of the one or more offsets in a header of the leaf block, the one offset of the one or more offsets indicating the offset for each key of the first subset of keys is the same size and is based on the ordinal key index.
11. The computer-implemented method of claim 10 , wherein the ordinal key index and the one offset of the one or more offsets implicitly represent each key of the third subset of keys.
12. The computer-implemented method of claim 7 , wherein the hierarchical data structure is a B+Tree, wherein each key of the subset of the set of keys represents a file offset of a file, and wherein the B+Tree is used to store a file layout tree representing the file.
13. A non-transitory machine-readable storage medium, including instructions configured to cause a processing apparatus to perform operations including:
accessing a set of keys, each key of the set of keys being associated with a pointer to one or more data blocks of a block-oriented storage device, the set of keys being arranged according to an ordinal key index, and the set of keys including a start key being first from amongst the set of keys;
defining a hierarchical data structure configured to store the set of keys, the hierarchical data structure including a leaf block, the leaf block being configured to store a first subset of keys of the set of keys according to the ordinal key index;
storing first one or more keys of the first subset of keys in the leaf block, the first one or more keys including at least the start key;
determining a set of offsets, each offset the set of offsets corresponding to a key of the first subset of keys;
determining whether the first subset of keys is contiguous;
in response to determining that the first subset of keys is contiguous, compressing the first subset of keys by
storing one or more first offsets of the set of offsets in the leaf block, the one or more first offsets corresponding to second one or more keys of the first subset of keys; and
in response to determining that the subset of the set of keys are not contiguous, compressing the first subset of keys by storing one or more second of the set of offsets in the leaf block in a new data structure including a set of records, the one or more second offsets being contiguous in the new data structure, and each record of the set of records of the new data structure including a key after the start key of the first subset of the set of keys and the associated offset of the key.
14. The non-transitory machine-readable storage medium of claim 13 , wherein determining the set of offsets further comprises comparing the start key with each other key of the first subset of keys, wherein each offset of the set of offsets corresponds to a difference between the start key and a subsequent key of the first subset of keys.
15. The non-transitory machine-readable storage medium of claim 14 , wherein each particular offset of the one or more offsets are stored:
as a partial keys corresponding to the difference between the start key and a subsequent key of the first subset of keys in the leaf block subsequent to the start key, according to the ordinal key index.
16. The non-transitory machine-readable storage medium of claim 13 , wherein the operation of compressing the first subset of the set of keys further comprises:
determining that each offset of the set of offsets has a same size; and
in response to determining that each offset of the set of offsets has a same size, storing one offset of the one or more offsets in a header of the leaf block, the one offset of the one or more offsets indicating the offset for each key of the first subset of keys is the same size and is based on the ordinal key index.
17. The non-transitory machine-readable storage medium of claim 16 , wherein the ordinal key index and the one offset of the one or more offsets implicitly represent each key of the third subset of keys.