Time-series data storage and processing database system
A database system is described that includes components for storing time-series data and executing custom, user-defined computational expressions in substantially real-time such that the results can be provided to a user device for display in an interactive user interface. For example, the database system may process stored time-series data in response to requests from a user device. The request may include a start time, an end time, a period, and/or a computational expression. The database system may retrieve the time-series data identified by the computational expression and, for each period, perform the arithmetic operation(s) identified by the computational expression on data values corresponding to times within the start time and the end time. Once all new data values have been generated, the database system may transmit the new data values to the user device for display in the interactive user interface.
1. A database configured to receive and process requests associated with time-series data and provide results to a user device, the database comprising:
a processing node comprising a computer processor and memory;
a time-series database storing a first data segment file and a second data segment file, wherein the first data segment file comprises data associated with a first time-series, wherein a size of the first data segment file is within a first size range, wherein a size of the second data segment file is within the first size range; and
a non-transitory computer readable storage medium storing program instructions, wherein the program instructions are configured for execution by the computer processor in order to cause the processing node to:
determine that a number of data segment files stored in the time-series database that comprise data associated with the first time-series and that have a size within the first size range is greater than a threshold value;
compact the first data segment file with the second data segment file to form a third data segment file, wherein a size of the third data segment file is within a second size range different than the first size range;
store the third data segment file in the time-series database;
determine that a number of data segment files stored in the time-series database that comprise data associated with the first time-series and that have a size within the second size range is greater than a second threshold value;
compact the third data segment file with other data segment files that comprise data associated with the first time-series and that have a size within the second size range to form a fourth data segment file, wherein a size of the fourth data segment file is within a third size range different than the first and second size ranges; and
store the fourth data segment file in the time-series database.
2. The database of claim 1 , wherein the threshold value and the second threshold value are the same value.
3. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to mark the first data segment file and the second data segment file for deletion.
4. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to:
receive second data associated with the first time-series;
compress the second data; and
store the compressed second data in the time-series database as the second data segment file.
5. The database of claim 4 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to compress the second data using one of block compression, out of box compression, or a delta-based compression.
6. The database of claim 4 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to store the compressed second data in a global write ahead log and in an in-memory buffer associated with the first time-series.
7. The database of claim 6 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to store the compressed second data in a local write ahead log associated with the first time-series in response to the global write ahead log reaching capacity.
8. The database of claim 7 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to store the compressed second data in the time-series database as the second data segment file in response to the local write ahead log reaching capacity.
9. The database of claim 1 , wherein a range of sizes corresponding to the second size range is larger than a range of sizes corresponding to the first size range.
10. The database of claim 1 , wherein the first size range is contiguous with the second size range.
11. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to:
receive a series expression, a start time, an end time, and a period from a user device;
retrieve, from the time-series database, a portion of the third data segment file corresponding to the start time and the end time; and
transmit the portion of the third data segment file to a plug-in computation device for generating, for each period between the start time and the end time, a value based on a computation identified by the series expression.
12. The database of claim 11 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to:
receive the generated values from the plug-in computation device; and
transmit the generated values to the user device for display.
13. The database of claim 11 , wherein the start time and the end time correspond to a window of data displayed in an interactive user interface generated by the user device.
14. The database of claim 13 , wherein the period identifies a period of time that corresponds with a width of a pixel in the interactive user interface.
15. The database of claim 1 , wherein the first data segment file stored in the time-series database is immutable.