Time-series data storage and processing database system
A database system is described that includes components for storing time-series data and executing custom, user-defined computational expressions in substantially real-time such that the results can be provided to a user device for display in an interactive user interface. For example, the database system may process stored time-series data in response to requests from a user device. The request may include a start time, an end time, a period, and/or a computational expression. The database system may retrieve the time-series data identified by the computational expression and, for each period, perform the arithmetic operation(s) identified by the computational expression on data values corresponding to times within the start time and the end time. Once all new data values have been generated, the database system may transmit the new data values to the user device for display in the interactive user interface.
1. A database configured to receive and process requests associated with a plurality of stored time-series data and provide results to a user device for display in an interactive user interface, the database comprising:
a discovery node comprising a first computer processor;
a processing node comprising a second computer processor and memory; and
a time-series database storing the plurality of time-series data, wherein the plurality of time-series data comprises first time-series data and second time-series data, and wherein the memory stores a copy of a portion of the plurality of time-series data; and
a computer readable storage medium storing first program instructions and second program instructions,
wherein the first program instructions are configured for execution by the first computer processor in order to cause the database to transmit a series expression, a start time, an end time, and a period received from the user device to the processing node, wherein the first time-series data and the second time-series data correspond to the series expression, and
wherein the second program instructions are configured for execution by the second computer processor in order to cause the database to:
retrieve, from the memory, a portion of the first time-series data and a portion of the second time-series data;
for each period between the start time and the end time,
identify a first data value from the portion of the first time-series data and a second data value from the portion of the second time-series data that are both associated with a same timestamp value, and
generate a value based on a computation identified by the series expression that is applied to the first data value and the second data value.
2. The database of claim 1 , wherein the second program instructions are further configured for execution by the second computer processor in order to cause the database to transmit the generated values to the user device for display in the interactive user interface.
3. The database of claim 1 , wherein the first program instructions are further configured for execution by the first computer processor in order to cause the database to determine that the processing node is associated with the first time-series data and the second time-series data.
4. The database of claim 1 , wherein the second program instructions are further configured for execution by the second computer processor in order to cause the database to retrieve, from the memory, data values from the first time-series data that are associated with timestamp values that fall within the start time and the end time and data values from the second time-series data that are associated with timestamp values that fall within the start time and the end time.
5. The database of claim 1 , wherein the first data value comprises a plurality of data values that are each associated with a different timestamp value, and wherein the second data value comprises a plurality of data values that are each associated with a different timestamp value.
6. The database of claim 1 , wherein the first program instructions are further configured for execution by the first computer processor in order to cause the database to:
receive, from a data source, third time-series data and fourth time-series data, wherein the third time-series data and the fourth time-series data correspond with a first sensor and comprise overlapping time values; and
compact the third time-series data and the fourth time-series data to generate the second time-series data.
7. The database of claim 6 , wherein the first program instructions are further configured for execution by the first computer processor in order to cause the database to:
determine, for each overlapping time value, whether a third data value corresponding to the third time-series data or a fourth data value corresponding to the fourth time-series data is stored in a later-modified file; and
insert the data value stored in the later-modified file into the second time-series data in association with the respective overlapping time value.
8. The database of claim 1 , wherein the start time and the end time correspond to a window of data viewed by a user via the interactive user interface.
9. The database of claim 1 , wherein the period identifies a period of time that corresponds with a width of a pixel in the interactive user interface.
10. The database of claim 9 , wherein the second program instructions are further configured for execution by the second computer processor in order to cause the database to:
determine whether the period is greater than a period of time between each timestamp value in the first time-series data;
aggregate data values such that a period of time between an earliest timestamp value corresponding to a data value in the aggregate and a latest timestamp value corresponding to a data value in the aggregate equals the period in response to a determination that the period is greater than the period of time between each timestamp value in the first time-series data; and
for each period between the start time and the end time,
identify a first aggregated data value from the portion of the first time-series data and the second data value from the portion of the second time-series data that are both associated with a same timestamp value, and
generate a value based on a computation identified by the series expression that is applied to the first aggregated data value and the second data value.
11. The database of claim 1 , wherein the series expression identifies a first operation associated with the first time-series data and the second time-series data and a second operation associated with a result of the first operation and the first time-series data.
12. The database of claim 1 , wherein the series expression comprises one of a sum, a difference, a product, a ratio, a moving average, a zScore, or a square root.
13. The database of claim 1 , wherein the first time-series data stored in the time-series database is immutable.