SYSTEMS AND METHOD FOR PROCESSING TIMESERIES DATA
In some implementations, events measured at various points in time may be organized in a data structure that defines an event represented by a document. In particular, events can be organized in columns of documents referred to as buckets. These buckets may be indexed using B-trees by addressing metadata values or value ranges. Buckets may be defined by periods of time. Documents may also be geoindexed and stored in one or more locations in a distributed computer network. One or more secondary indexes may be created based on time and/or metadata values within documents.
1 . A system comprising:
a database engine configured to store, in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to:
store, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents; and
index the plurality of timeseries events represented by the plurality of respective documents based on time values.
2 . The system according to claim 1 , wherein the database is a non-relational database comprising the plurality of documents.
3 . The system according to claim 1 , wherein the database engine is configured to index the plurality of documents using a B-tree.
4 . The system according to claim 1 , wherein the database engine is configured to store a time-based event that is represented by a single logical document.
5 . The system according to claim 1 , wherein the database engine is configured to index the plurality of documents.
6 . The system according to claim 5 , wherein the database engine is configured to index the plurality of documents based on a user-defined entity.
7 . The system according to claim 5 , wherein the user-defined entity includes metadata values within the plurality of documents, and wherein the metadata values include at least one of the group comprising a data source and a data region.
8 . The system according to claim 1 , wherein the database engine is further configured to create an on-demand materialized view of the plurality of documents.
9 . The system according to claim 7 , wherein the on-demand materialized view of the plurality of documents is an independent collection of data.
10 . The system according to claim 9 , wherein the independent collection of data is created within a pipeline processing stage using at least one pipeline operator.
11 . The system according to claim 1 , wherein each bucket of documents represents data collected at a particular moment of time.
12 . The system according to claim 1 , wherein the database engine is adapted to sample the database comprising the plurality of documents within buckets.
13 . The system according to claim 12 , wherein the bucket includes a group of measurements each having the same metadata over a limited period of time.
14 . The system according to claim 11 , wherein each bucket is indexed with a respective key.
15 . The system according to claim 12 , wherein the database engine is configured to perform a random sampling of buckets.
16 . The system according to claim 1 , wherein the database engine is configured to perform an unpacking of the bucket using a pipeline operator.
17 . The system according to claim 1 , wherein the database engine is configured to perform windowing operations using window bounds based on time and/or the plurality of documents.
18 . The system according to claim 17 , wherein the database engine is adapted to perform a windowing operation that produces an output stage that depends upon a range of input documents defined by the window bounds and a partition key.
19 . The system according to claim 13 , wherein the bucket defines a window of a predetermined amount of time.
20 . The system according to claim 19 , wherein at least one or more buckets associated with a plurality of windows are overlapping with respect to time.
21 . The system according to claim 1 , wherein the database engine is configured to index the plurality of time series events based on geographically-based indices.
22 . The system according to claim 1 , wherein the database engine is configured to archive data associated with a selected one or more buckets to a cold storage entity and delete, from a hot storage location, the selected one or more buckets.
23 . The system according to claim 1 , wherein the database engine is configured to archive data to a cold storage entity based on one or more parameters based on the documents.
24 . A method comprising acts of:
storing, by a database engine in a database, a plurality of timeseries events as a plurality of documents within a bucket, the database engine being further configured to perform acts of:
storing, in a columnar format, the plurality of timeseries events represented by the plurality of respective documents; and
indexing the plurality of timeseries events represented by the plurality of respective documents based on time values.
25 . The method according to claim 24 , wherein the database is a non-relational database comprising the plurality of documents.
26 . The method according to claim 24 , further comprising indexing the plurality of documents using a B-tree.
27 . The method according to claim 24 , wherein the database engine is configured to store a time-based event that is represented by a single logical document.
28 . The method according to claim 24 , further comprising an act of indexing, by the database engine, the plurality of documents based on a user-defined entity.
29 . The method according to claim 28 , wherein the user-defined entity includes metadata values within the plurality of documents, and wherein the metadata values include at least one of the group comprising a data source and a data region.
30 . The method according to claim 24 , further comprising an act of creating an on-demand materialized view of the plurality of documents.