Time-series data storage and processing database system
A database system is described that includes components for storing time-series data and executing custom, user-defined computational expressions in substantially real-time such that the results can be provided to a user device for display in an interactive user interface. For example, the database system may process stored time-series data in response to requests from a user device. The request may include a start time, an end time, a period, and/or a computational expression. The database system may retrieve the time-series data identified by the computational expression and, for each period, perform the arithmetic operation(s) identified by the computational expression on data values corresponding to times within the start time and the end time. Once all new data values have been generated, the database system may transmit the new data values to the user device for display in the interactive user interface.
1. A database comprising:
a computer processor;
memory configured to store a portion of first time-series data and a portion of second time-series data; and
a computer readable storage medium storing program instructions, wherein the program instructions are configured for execution by the computer processor in order to cause the database to:
obtain a series expression, a start time, and an end time provided by a user device via a network; and
for one or more periods between the start time and the end time,
identify whether any data value from the portion of the second time-series data has a same timestamp value as a first data value from the portion of the first time-series data,
perform interpolation on the portion of the second time-series data to estimate a second data value that has the same timestamp value as the first data value in response to a determination that no value from the portion of the second time-series data has the same timestamp value as the first data value,
apply, by the database, a computation identified by the series expression received from the user device to the first data value and the estimated second data value or to the first data value and a data value from the portion of the second time-series data that has the same timestamp value as the first data value to generate a value, and
transmit the value to the user device for display in an interactive user interface.
2. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the database to retrieve, from the memory, data values from the first time-series data that are associated with timestamp values that fall within the start time and the end time and data values from the second time-series data that are associated with timestamp values that fall within the start time and the end time.
3. The database of claim 1 , wherein the first data value comprises a plurality of data values that are each associated with a different timestamp value, and wherein a third data value from the portion of the second time-series data comprises a plurality of data values that are each associated with a different timestamp value.
4. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the database to:
receive, from a data source, third time-series data and fourth time-series data, wherein the third time-series data and the fourth time-series data correspond with a first sensor and comprise overlapping time values; and
compact the third time-series data and the fourth time-series data to generate the second time-series data.
5. The database of claim 4 , wherein the program instructions are further configured for execution by the computer processor in order to cause the database to:
determine, for each overlapping time value, whether a third data value corresponding to the third time-series data or a fourth data value corresponding to the fourth time-series data is stored in a later-modified file; and
insert the data value stored in the later-modified file into the second time-series data in association with the respective overlapping time value.
6. The database of claim 1 , wherein the start time and the end time correspond to a window of data viewed by a user via the interactive user interface.
7. The database of claim 1 , wherein a first period in the one or more periods identifies a period of time that corresponds with a width of a pixel in the interactive user interface.
8. The database of claim 7 , wherein the program instructions are further configured for execution by the computer processor in order to cause the database to:
determine whether the first period is greater than a period of time between each timestamp value in the first time-series data;
aggregate data values such that a period of time between an earliest timestamp value corresponding to a data value in the aggregate and a latest timestamp value corresponding to a data value in the aggregate equals the first period in response to a determination that the first period is greater than the period of time between each timestamp value in the first time-series data; and
for each period between the start time and the end time,
identify a first aggregated data value from the portion of the first time-series data and a third data value from the portion of the second time-series data that are both associated with a same timestamp value, and
apply the computation identified by the series expression to the first aggregated data value and the third data value to generate the value.
9. The database of claim 1 , wherein the series expression identifies a first operation associated with the first time-series data and the second time-series data and a second operation associated with a result of the first operation and the first time-series data.
10. The database of claim 1 , wherein the series expression comprises one of a sum, a difference, a product, a ratio, a moving average, a zScore, or a square root.
11. The database of claim 1 , wherein the first time-series data stored in the time-series database is immutable.
12. A method comprising:
as implemented by a database configured with specific computer-executable instructions,
obtaining a series expression, a start time, and an end time provided by a user device via a network; and
for one or more periods between the start time and the end time,
identifying whether any data value from a portion of a second time-series data stored in memory of the database has a same timestamp value as a first data value from a portion of a first time-series data stored in the memory,
performing interpolation on the portion of the second time-series data to estimate a second data value that has the same timestamp value as the first data value in response to a determination that no value from the portion of the second time-series data has the same timestamp value as the first data value,
applying, by the database, a computation identified by the series expression received from the user device to the first data value and the estimated second data value or to the first data value and a data value from the portion of the second time-series data that has the same timestamp value as the first data value to generate a value, and
transmitting the value to the user device for display in an interactive user interface.
13. The method of claim 12 , further comprising retrieving, from the memory, data values from the first time-series data that are associated with timestamp values that fall within the start time and the end time and data values from the second time-series data that are associated with timestamp values that fall within the start time and the end time.
14. The method of claim 12 , wherein the first data value comprises a plurality of data values that are each associated with a different timestamp value, and wherein a third data value from the portion of the second time-series data comprises a plurality of data values that are each associated with a different timestamp value.
15. The method of claim 12 , further comprising:
receiving, from a data source, third time-series data and fourth time-series data, wherein the third time-series data and the fourth time-series data correspond with a first sensor and comprise overlapping time values; and
compacting the third time-series data and the fourth time-series data to generate the second time-series data.
16. The method of claim 15 , further comprising:
determining, for each overlapping time value, whether a third data value corresponding to the third time-series data or a fourth data value corresponding to the fourth time-series data is stored in a later-modified file; and
inserting the data value stored in the later-modified file into the second time-series data in association with the respective overlapping time value.
17. The method of claim 12 , wherein the start time and the end time correspond to a window of data viewed by a user via the interactive user interface.
18. The method of claim 12 , wherein a first period in the one or more periods identifies a period of time that corresponds with a width of a pixel in the interactive user interface.
19. The method of claim 18 , further comprising:
determining whether the first period is greater than a period of time between each timestamp value in the first time-series data;
aggregating data values such that a period of time between an earliest timestamp value corresponding to a data value in the aggregate and a latest timestamp value corresponding to a data value in the aggregate equals the first period in response to a determination that the first period is greater than the period of time between each timestamp value in the first time-series data; and
for each period between the start time and the end time,
identifying a first aggregated data value from the portion of the first time-series data and a third data value from the portion of the second time-series data that are both associated with a same timestamp value, and
applying the computation identified by the series expression to the first aggregated data value and the third data value to generate the value.
20. Non-transitory, computer-readable storage media comprising computer-executable instructions for receiving and processing requests associated with a plurality of stored time-series data, wherein the computer-executable instructions, when executed by a database, cause the database to:
obtain a series expression, a start time, and an end time provided by a user device via a network; and
for one or more periods between the start time and the end time,
identify whether any data value from a portion of a second time-series data stored in memory of the database has a same timestamp value as a first data value from a portion of a first time-series data,
perform interpolation on the portion of the second time-series data to estimate a second data value that has the same timestamp value as the first data value in response to a determination that no value from the portion of the second time-series data has the same timestamp value as the first data value,
apply, by the database, a computation identified by the series expression received from the user device to the first data value and the estimated second data value or to the first data value and a data value from the portion of the second time-series data that has the same timestamp value as the first data value to generate a value, and
transmit the value to the user device for display in an interactive user interface.