Systems and methods for processing timeseries data
A system is provided for storing, in a database, a plurality of timeseries represented by a plurality of respective documents events in a columnar format. The system further is adapted compress at least one of the values within the plurality of documents. According to some embodiments, the system stores the compressed values as a Simple-8b block and calculates the optimal Simple-8b selector. According to some embodiments, the system is adapted to determine a secondary index based on values within the bucket.
1. A system comprising:
a database engine in a database executed by at least one computer processor and configured to store timeseries events, the database engine being further configured to:
receive, over a network operatively coupled to the database engine, the timeseries events as one or more BSON documents, wherein the one or more BSON documents includes timeseries measurements;
compress the one or more BSON documents, wherein the compressing comprises:
performing a delta compression operation or a delta-of-delta compression operation on at least one of the timeseries measurements; and
store, in one or more storage entities of the database, the compressed one or more BSON documents using a Simple-8b block, wherein the database engine is configured to store the compressed one or more BSON documents using the Simple-8b block by:
determining whether an optimal Simple-8b selector to be used is an extended Simple-8b selector based on a number of trailing zeros in one or more of the compressed measurements; and
calculating the optimal Simple-8b selector for the Simple-8b block based on the compressed measurements and the determination of whether the optimal Simple-8b selector is the extended Simple-8b selector; and
storing the compressed measurements in the one or more storage entities of the database using the Simple-8b block employing the optimal Simple-8b selector.
2. The system according to claim 1 , wherein the compressing further comprises performing a ZigZag encoding using an output of the delta compression or the delta-of-delta compression.
3. The system according to claim 2 , wherein the compressing further comprises calculating a number associated with meaningful bits of an output of the ZigZag encoding.
4. The system according to claim 3 , wherein the compressing further comprises performing a scale encoding on the at least one of the timeseries measurements.
5. The system according to claim 1 , wherein the delta compression operation or a delta-of-delta compression operation of a document element based on a type value of the document element.
6. The system according to claim 1 , wherein the storing of the compressed one or more BSON documents further comprises:
storing a first portion of the compressed measurements using the Simple-8b block, and
storing a second portion of the compressed measurements using a second Simple-8b block.
7. The system according to claim 1 , wherein the Simple-8b block comprises
the Simple-8b selector, the Simple-8b selector configured to encode a number of slots for storing the compressed measurements; and
a shift configured to encode the number of trailing zeros stored within a slot of the Simple-8b block.
8. A method comprising:
storing, by a database engine in a database, timeseries events, the database engine being configured to:
receive, over a network operatively coupled to the database engine, the timeseries events as one or more BSON documents, wherein the one or more BSON documents includes timeseries measurements;
compress the one or more BSON documents, wherein the compressing comprises:
performing a delta compression operation or a delta-of-delta compression operation on at least one of the timeseries measurements; and
storing, in one or more storage entities of the database, the compressed one or more BSON documents using a Simple-8b block, wherein the storing comprises:
determining whether an optimal Simple-8b selector to be used in an extended Simple-8b selector based on a number of trailing zeros in one or more of the compressed measurements; and
calculating the optimal Simple-8b selector for the Simple-8b block based on the compressed measurements and the determination of whether the optimal Simple-8b selector is the extended Simple-8b selector; and
storing the compressed measurements in the one or more storage entities of the database using the Simple-8b block employing the optimal Simple-8b selector.
9. The method according to claim 8 , wherein the compressing further comprises performing a ZigZag encoding using an output of the delta compression or the delta-of-delta compression.
10. The method according to claim 9 , wherein the compressing further comprises calculating a number associated with meaningful bits of an output of the ZigZag encoding.
11. The method according to claim 10 , wherein the compressing further comprises performing a scale encoding on the at least one of the timeseries measurements.
12. The method according to claim 8 , wherein the delta compression operation or a delta-of-delta compression operation of a document element based on a type value of the document element.
13. The method according to claim 8 , wherein the storing of the compressed one or more BSON documents further comprises:
storing a first portion of the compressed measurements using the Simple-8b block, and
storing a second portion of the compressed measurements using a second Simple-8b block.
14. The method according to claim 8 , wherein the Simple-8b block comprises
the Simple-8b selector, the Simple-8b selector configured to encode a number of slots for storing the compressed measurements; and
a shift configured to encode the number of trailing zeros stored within a slot of the Simple-8b block.
15. A non-transitory computer-readable medium containing instruction that, when executed, cause at least one computer hardware processor to perform:
storing, in a database, timeseries events, the at least one computer hardware processor being further configured to:
receive, over a network operatively coupled to the database, the timeseries events as one or more BSON documents, wherein the one or more BSON documents includes timeseries measurements;
compress the one or more BSON documents, wherein the compressing comprises:
performing a delta compression operation or a delta-of-delta compression operation on at least one of the timeseries measurements; and
storing, in one or more storage entities of the database, the compressed one or more BSON documents using a Simple-8b block, wherein the at least one computer hardware processor is configured to store the compressed one or more BSON documents using the Simple-8b block by:
determining whether an optimal Simple-8b selector to be used in an extended Simple-8b selector based on a number of trailing zeros in one or more of the compressed measurements; and
calculating the optimal Simple-8b selector for the Simple-8b block based on the compressed measurements and the determination of whether the optimal Simple-8b selector is the extended Simple-8b selector; and
storing the compressed measurements in the one or more storage entities of the database using the Simple-8b block employing the optimal Simple-8b selector.
16. The non-transitory computer-readable medium according to claim 15 , wherein the compressing further comprises performing a ZigZag encoding using an output of the delta compression or the delta-of-delta compression.
17. The non-transitory computer-readable medium according to claim 16 , wherein the compressing further comprises calculating a number associated with meaningful bits of an output of the ZigZag encoding.
18. The non-transitory computer-readable medium according to claim 17 , wherein the compressing further comprises performing a scale encoding on the at least one of the timeseries measurements.
19. The non-transitory computer-readable medium according to claim 15 , wherein the storing of the compressed one or more BSON documents further comprises:
storing a first portion of the compressed measurements using the Simple-8b block, and
storing a second portion of the compressed measurements using a second Simple-8b block.
20. The non-transitory computer-readable medium according to claim 15 , wherein the Simple-8b block comprises
the Simple-8b selector, the Simple-8b selector configured to encode a number of slots for storing the compressed measurements; and
a shift configured to encode a number of trailing zeros stored within a slot of the Simple-8b block.