Systems and methods for processing timeseries data
A system is provided for storing, in a database, a plurality of timeseries represented by a plurality of respective documents events in a columnar format. The system further is adapted compress at least one of the values within the plurality of documents. According to some embodiments, the system stores the compressed values as a Simple-8b block and calculates the optimal Simple-8b selector. According to some embodiments, the system is adapted to determine a secondary index based on values within the bucket.
1 . A system comprising:
a database engine configured to store, in an unstructured database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to:
determine one or more output documents comprising a subset of the plurality of timeseries events based at least in part on a first window bound and a second window bound, the first and second window bounds indicative of a range of time associated with one or more documents of the plurality of documents comprising the subset of the plurality of timeseries events, wherein:
the first and second window bounds are determined based at least in part on a time associated with a first timeseries event;
the first window bound is indicative of a start of the range of time, the start being prior to the time associated with the first timeseries event of the plurality of timeseries events; and
the second window bound is indicative of an end of the range of time, the end following the time associated with the first timeseries event of the plurality of timeseries events;
provide the one or more output documents as output of the database engine; and
store, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, wherein the act of storing includes compressing at least one of a series of data values among the plurality of output documents within the bucket, the bucket configured to store documents within the range of time indicated by the first and second window bounds.
2 . The system according to claim 1 , wherein the database engine is configured to perform a delta compression among data values associated with different timeseries measurements from the plurality of documents within the bucket.
3 . The system according to claim 1 , wherein the database engine is configured to perform a transformation of a document element to a signed integer.
4 . The system according to claim 1 , wherein the database engine is configured to perform a delta compression operation or a delta-of-delta compression operation of a document element based on a type value of the document element.
5 . The system according to claim 4 , wherein the database engine is configured to perform a ZigZag encoding operation using an output of the performed delta compression operation or delta-of-delta compression operation.
6 . The system according to claim 5 , wherein the database engine is configured to perform a Simple-8b encoding operation using an output of the zigzag encoding operation.
7 . The system according to claim 6 , wherein the database engine is adapted to change a scale encoding of the data values responsive to receiving new timeseries event data.
8 . The system according to claim 1 , wherein the plurality of documents includes one or more BSON documents.
9 . The system according to claim 1 , wherein the database engine is configured to index the plurality of timeseries events represented by the plurality of respective documents based on time values.
10 . The system according to claim 1 , wherein the database is a non-relational database comprising the plurality of documents.
11 . The system according to claim 1 , wherein the database engine is configured to store a time-based event that is represented by a single logical document.
12 . The system according to claim 1 , wherein:
the plurality of documents includes a plurality of measurement fields,
the columnar format is configured to store interleaved data associated with the plurality of measurement fields.
13 . The system of claim 12 , wherein the columnar format is further configured to store a control byte, the control byte is configured to differentiate between uncompressed and compressed elements.
14 . The system of claim 1 , wherein the database engine is configured to perform a run-length encoding operation repeated data values.
15 . A method comprising:
storing, by a database engine in an unstructured database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to perform acts of:
determining one or more output documents comprising a subset of the plurality of timeseries events based at least in part on a first window bound and a second window bound, the first and second window bounds indicative of a range of time associated with one or more documents of the plurality of documents comprising the subset of the plurality of timeseries events, wherein:
the first and second window bounds are determined based at least in part on a time associated with a first timeseries event:
the first window bound is indicative of a start of the range of time, the start being prior to the time associated with the first timeseries event of the plurality of timeseries events; and
the second window bound is indicative of an end of the range of time, the end following the time associated with the first timeseries event of the plurality of timeseries events;
providing the one or more output documents as output of the database engine; and
storing, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, wherein the act of storing includes compressing at least one of a series of data values among the plurality of documents within the bucket, the bucket configured to store documents within the range of time indicated by the first and second window bounds.
16 . The method according to claim 15 , wherein the database engine is configured to perform a delta compression operation or a delta-of-delta compression operation of a document element based on a type value of the document element.
17 . The method according to claim 16 , wherein the database engine is configured to perform a zigzag encoding operation using an output of the performed delta compression operation or delta-of-delta compression operation.
18 . The method according to claim 17 , wherein the database engine is configured to perform a Simple-8b encoding operation using an output of the zigzag encoding operation.
19 . A non-transitory computer-readable medium containing instruction that, when executed, cause at least one computer hardware processor to perform:
storing, in an unstructured database using a database engine executed by the at least one computer hardware processor, a plurality of timeseries events as a plurality of documents within a bucket, the at least one computer hardware processor being further configured to:
determine one or more output documents comprising a subset of the plurality of timeseries events based at least in part on a first window bound and a second window bound, the first and second window bounds indicative of a range of time associated with one or more documents of the plurality of documents comprising the subset of the plurality of timeseries events, wherein:
the first and second window bounds are determined based at least in part on a time associated with a first timeseries event;
the first window bound is indicative of a start of the range of time, the start being prior to the time associated with the first timeseries event of the plurality of timeseries events; and
the second window bound is indicative of an end of the range of time, the end following the time associated with the first timeseries event of the plurality of timeseries events;
provide the one or more output documents as output of the database engine; and
store, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents, wherein the act of storing includes compressing at least one of a series of data values among the plurality of documents within the bucket, the bucket configured to store documents within the range of time indicated by the first and second window bounds.
20 . The non-transitory computer-readable medium according to claim 19 , wherein the at least one computer hardware processor is further configured to perform a delta compression operation or a delta-of-delta compression operation of a document element based on a type value of the document element.