Time-series data storage and processing database system
A database system is described that includes components for storing time-series data and executing custom, user-defined computational expressions in substantially real-time such that the results can be provided to a user device for display in an interactive user interface. For example, the database system may process stored time-series data in response to requests from a user device. The request may include a start time, an end time, a period, and/or a computational expression. The database system may retrieve the time-series data identified by the computational expression and, for each period, perform the arithmetic operation(s) identified by the computational expression on data values corresponding to times within the start time and the end time. Once all new data values have been generated, the database system may transmit the new data values to the user device for display in the interactive user interface.
1. A database comprising:
one or more data stores configured to store time series data; and
one or more processing nodes comprising respective memories, memory maps, and computer processors,
wherein the database is configured to:
obtain an operation, a start time, and an end time provided by a user device via a network;
determine a processing node, of the one or more processing nodes, based at least in part on the obtained operation;
access, by the determined processing node and by reference to the memory map of the determined processing node, a portion of first time-series data and a portion of second time-series data from the memory of the determined processing node;
determine that a timestamp value from the portion of first time-series data does not align with a timestamp value from the portion of second time-series data;
estimate, by the determined processing node, a missing data value using the portion of the second time-series data;
execute, by the determined processing node, the operation using the portion of the first time-series data and the missing data value to generate an output; and
transmit the output to the user device,
wherein the memory map of the determined processing node identifies segments of the memory of the determined processing node that store portions of time series data, and wherein the stored portions of time series data are obtained by mapping from a data store of the one or more data stores that is associated with the determined processing node.
2. The database of claim 1 , wherein the database is further configured to retrieve data values from the first time-series data that are associated with timestamp values that fall within the start time and the end time and data values from the second time-series data that are associated with timestamp values that fall within the start time and the end time.
3. The database of claim 1 , wherein a first data value from the portion of the first time-series data comprises a plurality of data values that are each associated with a different timestamp value, and wherein a second data value from the portion of the second time-series data comprises a plurality of data values that are each associated with a different timestamp value.
4. The database of claim 1 , wherein the database is further configured to:
receive, from a data source, third time-series data and fourth time-series data, wherein the third time-series data and the fourth time-series data correspond with a first sensor and comprise overlapping time values; and
compact the third time-series data and the fourth time-series data to generate the second time-series data.
5. The database of claim 4 , wherein the database is further configured to:
determine, for each overlapping time value, whether a third data value corresponding to the third time-series data or a fourth data value corresponding to the fourth time-series data is stored in a later-modified file; and
insert the data value stored in the later-modified file into the second time-series data in association with the respective overlapping time value.
6. The database of claim 1 , wherein the start time and the end time correspond to a window of data viewed by a user via an interactive user interface.
7. The database of claim 1 , wherein a period between the start time and the end time identifies a period of time that corresponds with a width of a pixel in an interactive user interface.
8. The database of claim 7 , wherein the database is further configured to:
determine whether the period is greater than a period of time between each timestamp value in the first time-series data;
aggregate data values such that a period of time between an earliest timestamp value corresponding to a data value in the aggregate and a latest timestamp value corresponding to a data value in the aggregate equals the period in response to a determination that the period is greater than the period of time between each timestamp value in the first time-series data; and
for the period,
identify a first aggregated data value from the portion of the first time-series data and a third data value from the portion of the second time-series data that are both associated with a same timestamp value, and
apply the operation to the first aggregated data value and the third data value to generate the value.
9. The database of claim 1 , wherein the operation comprises a first operation associated with the first time-series data and the second time-series data and a second operation associated with a result of the first operation and the first time-series data.
10. The database of claim 1 , wherein the operation comprises one of a sum, a difference, a product, a ratio, a moving average, a zScore, or a square root.
11. The database of claim 1 , wherein the first time-series data is stored in the database and is immutable.
12. A method comprising:
obtaining, by a database configured with specific computer-executable instructions, an operation, a start time, and an end time provided by a user device via a network, wherein the database comprises:
one or more data stores configured to store time series data; and
one or more processing nodes comprising respective memories and memory maps;
determining a processing node, of the one or more processing nodes, based at least in part on the obtained operation;
accessing, by the determined processing node and by reference to the memory map of the determined processing node, a portion of first time-series data and a portion of second time-series data from the memory of the determined processing node;
determining, by the database, that a timestamp value from the portion of first time-series data does not align with a timestamp value from the portion of second time-series data;
estimating, by the determined processing node, a missing data value using the portion of the second time-series data;
executing, by the determined processing node, the operation using the portion of the first time-series data and the missing data value to generate an output; and
transmitting, by the database, the output to the user device,
wherein the memory map of the determined processing node identifies segments of the memory of the determined processing node that store portions of time series data, wherein the stored portions of time series data are obtained by mapping from a data store of the one or more data stores that is associated with the determined processing node, and wherein the method is performed by the database.
13. The method of claim 12 , further comprising retrieving data values from the first time-series data that are associated with timestamp values that fall within the start time and the end time and data values from the second time-series data that are associated with timestamp values that fall within the start time and the end time.
14. The method of claim 12 , wherein a first data value from the portion of the first time-series data comprises a plurality of data values that are each associated with a different timestamp value, and wherein a second data value from the portion of the second time-series data comprises a plurality of data values that are each associated with a different timestamp value.
15. The method of claim 12 , further comprising:
receiving, from a data source, third time-series data and fourth time-series data, wherein the third time-series data and the fourth time-series data correspond with a first sensor and comprise overlapping time values; and
compacting the third time-series data and the fourth time-series data to generate the second time-series data.
16. The method of claim 15 , further comprising:
determining, for each overlapping time value, whether a third data value corresponding to the third time-series data or a fourth data value corresponding to the fourth time-series data is stored in a later-modified file; and
inserting the data value stored in the later-modified file into the second time-series data in association with the respective overlapping time value.
17. The method of claim 12 , wherein the start time and the end time correspond to a window of data viewed by a user via an interactive user interface.
18. The method of claim 12 , wherein a period between the start time and the end time identifies a period of time that corresponds with a width of a pixel in an interactive user interface.
19. The method of claim 18 , further comprising:
determining whether first period is greater than a period of time between each timestamp value in the first time-series data;
aggregating data values such that a period of time between an earliest timestamp value corresponding to a data value in the aggregate and a latest timestamp value corresponding to a data value in the aggregate equals the first period in response to a determination that the period is greater than the period of time between each timestamp value in the first time-series data; and
for the period,
identifying a first aggregated data value from the portion of the first time-series data and a third data value from the portion of the second time-series data that are both associated with a same timestamp value, and
applying the operation to the first aggregated data value and the third data value to generate the value.
20. Non-transitory, computer-readable storage media comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a database comprising:
one or more data stores configured to store time series data; and
one or more processing nodes comprising respective memories, memory maps, and computer processors,
cause the database to:
obtain an operation, a start time, and an end time provided by a user device via a network;
determine a processing node, of the one or more processing nodes, based at least in part on the obtained operation;
access, by the determined processing node and by reference to the memory map of the determined processing node, a portion of first time-series data and a portion of second time-series data from the memory of the determined processing node;
determine that a timestamp value from the portion of first time-series data does not align with a timestamp value from the portion of second time-series data;
estimate, by the determined processing node, a missing data value using the portion of the second time-series data;
execute, by the determined processing node, the operation using the portion of the first time-series data and the missing data value to generate an output; and
transmit the output to the user device,
wherein the memory map of the determined processing node identifies segments of the memory of the determined processing node that store portions of time series data, and wherein the stored portions of time series data are obtained by mapping from a data store of the one or more data stores that is associated with the determined processing node.