Rule based metadata partitioning
Rule based metadata partitioning may be used for metadata from a distributed database. The partitioning is based on rules for different buckets. The rules may be defined based on a defined set of syntax or using a predefined rule template. The partitioning may include dynamic adjustments and include auto detection for access patterns that are used for updating or creating new rules.
1 . A method for metadata partitioning comprising:
accessing rules for the metadata partitioning for storage in distributed storage nodes, wherein the rules are automatically created based on a set of syntax and comprises both bucketing rules and sub-partition rules;
utilizing a predefined rule template associated with the rules, the predefined rule template being automatically recognized when bucketizing objects;
adjusting, dynamically, existing object metadata partitioning;
detecting, automatically, access patterns from client requests for improving the rules and for automatically creating new rules for the metadata partitioning, the new rules being created at least to overcome a scalability limitation of a single node in the distributed storage node for storing one partition and to improve query performance in reading metadata across a range of keys; and
storing the metadata partitioning in the distributed storage nodes.
2 . The method of claim 1 , wherein the rules are defined for different buckets.
3 . The method of claim 2 , wherein the set of syntax is used to create rules.
4 . The method of claim 1 , wherein the rules identify characteristic of an object hierarchy, and describe a storage of the metadata in different partitions.
5 . The method of claim 1 , wherein the predefined rule template is utilized for certain applications.
6 . The method of claim 1 , wherein an object metadata storage partition is adjusted based on one of the rules.
7 . A system comprising a memory for storing instructions and at least one processor configured to execute the instructions to:
access rules for metadata partitioning for storage in distributed storage nodes, wherein the rules are automatically created based on a set of syntax and comprises both bucketing rules and sub-partition rules;
utilize a predefined rule template associated with the rules, the predefined rule template being automatically recognized when bucketizing objects;
adjust, dynamically, existing object metadata partitioning;
detect, automatically, access patterns from client requests for improving the rules and for automatically creating new rules, the new rules being created at least to overcome a scalability limitation of a single node in the distributed storage node for storing one partition and to improve query performance in reading metadata across a range of keys; and
store the metadata partitioning in the distributed storage nodes.
8 . The system of claim 7 , wherein the rules are defined for different buckets.
9 . The system of claim 8 , wherein the set of syntax is used to create rules.
10 . The system of claim 7 , wherein the rules identify characteristic of an object hierarchy, and describe a storage of the metadata in different partitions.
11 . The system of claim 7 , wherein the predefined rule template is utilized for certain applications.
12 . The system of claim 7 , wherein an object metadata storage partition is adjusted based on one of the rules.