Time-series data storage and processing database system
A database system is described that includes components for storing time-series data and executing custom, user-defined computational expressions in substantially real-time such that the results can be provided to a user device for display in an interactive user interface. For example, the database system may process stored time-series data in response to requests from a user device. The request may include a start time, an end time, a period, and/or a computational expression. The database system may retrieve the time-series data identified by the computational expression and, for each period, perform the arithmetic operation(s) identified by the computational expression on data values corresponding to times within the start time and the end time. Once all new data values have been generated, the database system may transmit the new data values to the user device for display in the interactive user interface.
1. A database comprising:
a processing node comprising a computer processor and memory; and
a computer readable storage medium storing program instructions, wherein the program instructions are configured for execution by the computer processor in order to cause the processing node to:
determine that a number of data files stored in a time-series data store that have a size within a first size range is greater than a threshold value, wherein the time-series data store comprises a first data file and a second data file that each have a size within the first size range;
compact the first data file with the second data file to form a third data file, wherein a size of the third data file is within a second size range different than the first size range; and
compact the third data file with other data files that have a size within the second size range to form a fourth data file in response to a determination that a number of data files stored in the time-series data store that have a size within the second size range is greater than a second threshold value, and wherein a size of the fourth data file is within a third size range different than the first and second size ranges.
2. The database of claim 1 , wherein the threshold value and the second threshold value are the same value.
3. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to mark the first data file and the second data file for deletion.
4. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to:
receive first data;
compress the first data; and
store the compressed first data in the time-series data store as the second data file.
5. The database of claim 1 , wherein a range of sizes corresponding to the second size range is larger than a range of sizes corresponding to the first size range.
6. The database of claim 1 , wherein the first size range is contiguous with the second size range.
7. The database of claim 1 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to:
receive a series expression, a start time, an end time, and a period;
retrieve a portion of the third data file corresponding to the start time and the end time; and
transmit the portion of the third data file to a plug-in computation device for generating, for each period between the start time and the end time, a value based on a computation identified by the series expression.
8. The database of claim 1 , wherein the first data segment file stored in the time-series data store is immutable.
9. The database of claim 4 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to compress the first data using one of block compression, out of box compression, or a delta-based compression.
10. The database of claim 4 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to store the compressed first data in a global write ahead log and in an in-memory buffer.
11. The database of claim 10 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to store the compressed first data in a local write ahead log in response to the global write ahead log reaching capacity.
12. The database of claim 11 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to store the compressed first data in the time-series data store as the second data file in response to the local write ahead log reaching capacity.
13. The database of claim 7 , wherein the start time and the end time correspond to a window of data displayed in an interactive user interface generated by a user device.
14. The database of claim 13 , wherein the period identifies a period of time that corresponds with a width of a pixel in the interactive user interface.
15. The database of claim 7 , wherein the program instructions are further configured for execution by the computer processor in order to cause the processing node to:
receive the generated values from the plug-in computation device; and
transmit the generated values to a user device for display.
16. A computer-implemented method comprising:
determining, by a processing node of a database that comprises a computer processor and memory, that a number of data files stored in a time-series data store that have a size within a first size range is greater than a threshold value, wherein the time-series data store comprises a first data file and a second data file that each have a size within the first size range;
compacting the first data file with the second data file to form a third data file, wherein a size of the third data file is within a second size range different than the first size range; and
compacting the third data file with other data files that have a size within the second size range to form a fourth data file in response to a determination that a number of data files stored in the time-series data store that have a size within the second size range is greater than a second threshold value, and wherein a size of the fourth data file is within a third size range different than the first and second size ranges.
17. The computer-implemented method of claim 16 , wherein the threshold value and the second threshold value are the same value.
18. The computer-implemented method of claim 16 , further comprising:
receiving first data;
compressing the first data; and
storing the compressed first data in the time-series data store as the second data file.
19. The computer-implemented method of claim 16 , further comprising:
receiving a series expression, a start time, an end time, and a period;
retrieving a portion of the third data file corresponding to the start time and the end time; and
transmitting the portion of the third data file to a plug-in computation device for generating, for each period between the start time and the end time, a value based on a computation identified by the series expression.
20. A non-transitory computer-readable medium comprising one or more program instructions recorded thereon, the instructions configured for execution by a processing node of a database that comprises one or more processors in order to cause the database to:
determine that a number of data files stored in a time-series data store that have a size within a first size range is greater than a threshold value, wherein the time-series data store comprises a first data file and a second data file that each have a size within the first size range;
compact the first data file with the second data file to form a third data file, wherein a size of the third data file is within a second size range different than the first size range; and
compact the third data file with other data files that have a size within the second size range to form a fourth data file in response to a determination that a number of data files stored in the time-series data store that have a size within the second size range is greater than a second threshold value, and wherein a size of the fourth data file is within a third size range different than the first and second size ranges.