Systems and method for processing timeseries data
In some implementations, events measured at various points in time may be organized in a data structure that defines an event represented by a document. In particular, events can be organized in columns of documents referred to as buckets. These buckets may be indexed using B-trees by addressing metadata values or value ranges. Buckets may be defined by periods of time. Documents may also be geoindexed and stored in one or more locations in a distributed computer network. One or more secondary indexes may be created based on time and/or metadata values within documents.
1. A system comprising:
at least one processor configured to execute a database engine configured to:
store, in a database, a plurality of timeseries events as a plurality of documents within a bucket in a columnar format, each of the plurality of documents comprising a time value for one of the plurality of timeseries events and metadata associating the document with a common source of the plurality of timeseries events,
wherein the bucket stores the time values of the plurality of documents in the columnar format and the bucket further stores common metadata that is common to the plurality of documents, the common metadata associating the plurality of documents with the common source of the plurality of time series events; and
perform an unpacking of the bucket using a pipeline operator.
2. The system according to claim 1 , wherein the database engine is configured to unpack one or more of the plurality of timeseries events from the bucket.
3. The system according to claim 1 , wherein the database engine is configured to identify one or more buckets of a data collection identified in the database to unpack.
4. The system according to claim 2 , wherein the database engine is configured to unpack the one or more of the plurality of timeseries events from the bucket one event at a time.
5. The system according to claim 1 , wherein the database engine is configured to inspect a top-level data region of the bucket based on field names and wherein the database engine is further configured to construct a list of events to unpack from the bucket.
6. The system according to claim 1 , wherein the database engine is configured to store a time-based event that is represented by a single logical document.
7. The system according to claim 1 , wherein the database engine is configured to create an on-demand materialized view of the plurality of documents responsive to an unpacking event.
8. The system according to claim 7 , wherein the on-demand materialized view of the plurality of documents is an independent collection of data.
9. The system according to claim 8 , wherein the independent collection of data is created within a pipeline processing stage using at least one pipeline operator.
10. A method comprising acts of:
executing, by at least one processor, a database engine, the executing comprising:
storing, by the database engine in a database a plurality of timeseries events as a plurality of documents within a bucket in a columnar format, each of the plurality of documents comprising a time value for one of the plurality of timeseries events and metadata associating the document with a common source of the plurality of timeseries events,
wherein the bucket stores the time values of the plurality of documents in the columnar format and the bucket further stores common metadata that is common to the plurality of documents, the common metadata associating the plurality of documents with the common source of the plurality of time series events; and
performing an unpacking of the bucket using a pipeline operator.
11. The method according to claim 10 , further comprising an act of unpacking one or more of the plurality of timeseries events from the bucket.
12. The method according to claim 10 , further comprising an act of identifying one or more buckets of a data collection identified in the database to unpack.
13. The method according to claim 11 , further comprising an act of unpacking the one or more of the plurality of timeseries events from the bucket one event at a time.
14. The method according to claim 10 , further comprising an act of inspecting a top-level data region of the bucket based on field names and constructing a list of events to unpack from the bucket.
15. The method according to claim 10 , further comprising an act of storing a time-based event that is represented by a single logical document.
16. The method according to claim 10 , further comprising an act of creating an on-demand materialized view of the plurality of documents responsive to an unpacking event.
17. The method according to claim 16 , wherein the on-demand materialized view of the plurality of documents is an independent collection of data.
18. The method according to claim 17 , wherein the independent collection of data is created within a pipeline processing stage using at least one pipeline operator.
19. The system according to claim 1 , wherein the database comprises a flexible schema database and the metadata comprises a key-value pair.
20. The method according to claim 10 , wherein the database comprises a flexible schema database and the metadata comprises a key-value pair.
21. A database system, comprising:
a database; and
at least one processor configured to store timestamped measurements in the database using a bucket data structure at least in part by:
storing, in the bucket data structure, time values of a plurality of documents in a column, each of the plurality of documents representing a timestamped measurement of the timestamped measurements and comprising a time value of the time values and a metadata value; and
storing, in the bucket data structure, a common metadata value that matches the metadata value of the plurality of documents, wherein the common metadata value associates the plurality of documents with a source that generated the timestamped measurements represented by the plurality of documents,
wherein the at least one processor is further configured to unpack the bucket data structure using a pipeline operator.
22. The database system of claim 21 , wherein the bucket data structure comprises a plurality of columns storing data values of the plurality of documents, and the common metadata value is stored outside of the plurality of columns.