Data ingestion stager for time series database
View Patent ↗A time series data stager that receives input data sets and outputs output data blocks for ingestion into a time series database, with the out data blocks being sent at timings according to a sliding window based on a predetermined time.
1. A method comprising:
receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data;
for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time;
for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets;
for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system;
receiving, from a client query driver, by the time series stager system and at a first point in time, a first query;
performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results; and
outputting, by the time series stager and to the client query driver, the first query results.
2. The method of claim 1 wherein:
the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system.
3. The method of claim 1
the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system.
4. The method of claim 1 wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system.
5. The method of claim 1 further comprising:
maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system;
wherein the performance of the first query includes checking the index to determine the first query results.
6. The method of claim 1 wherein:
the data storage device includes a non-volatile data storage medium; and
the storage of the plurality of input data sets is non-volatile type data storage.
7. A computer program product (CPP) comprising:
a computer readable storage medium; and
computer code stored on the computer readable storage medium, the computer code including instructions and data for causing a processor(s) set to perform at least the following operations:
receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data,
for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time,
for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets,
for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system,
receiving, from a client query driver, by the time series stager system and at a first point in time, a first query,
performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results, and
outputting, by the time series stager and to the client query driver, the first query results.
8. The CPP of claim 7 wherein the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system.
9. The CPP of claim 7 wherein the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system.
10. The CPP of claim 7 wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system.
11. The CPP of claim 7 further comprising:
maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system;
wherein the performance of the first query includes checking the index to determine the first query results.
12. The CPP of claim 7 wherein:
the data storage device includes a non-volatile data storage medium; and
the storage of the plurality of input data sets is non-volatile type data storage.
13. A computer system (CS) comprising:
a processor(s) set;
a computer readable storage medium; and
computer code stored on the computer readable storage medium, the computer code including instructions and data for causing the processor(s) set to perform at least the following operations:
receiving, from a set of client data driver(s) and by a time series stager system, a plurality of input data sets designed for ingestion into a time series database, with each input data set including payload data and a timestamp related to collection of the payload data,
for each given input data set of the plurality of input data sets, storing the given input data set a data storage device in the time series stager system for at least a predetermined interval of time,
for each given input data set of the plurality of input data sets, outputting the given input data set to a time series database system in an order, relative to other input data sets of the plurality of input data sets, determined by respective timestamps of the plurality of input data sets,
for each given input data set of the plurality of input data sets, responsive to outputting of the given input data set, ceasing to store the given input data set in the data storage device of the time series stager system,
receiving, from a client query driver, by the time series stager system and at a first point in time, a first query,
performing the first query against input data sets stored in the data storage device of the time series stager system at the first point in time to yield first query results, and
outputting, by the time series stager and to the client query driver, the first query results.
14. The CS of claim 13 wherein the receipt of the plurality of input data sets occurs through an ingestion thread operating in a processor(s) set of the time series stager system.
15. The CS of claim 13 wherein the outputting of the input data sets occurs through a shipping thread operating in a processor(s) set of the time series stager system.
16. The CS of claim 13 wherein the outputting of the first query results occurs through a query thread operating in a processor(s) set of the time series stager system.
17. The CS of claim 13 further comprising:
maintaining an index of current input data sets of the plurality of input data sets that are currently stored in the data storage device of the time series stager system;
wherein the performance of the first query includes checking the index to determine the first query results.
18. The CS of claim 13 wherein:
the data storage device includes a non-volatile data storage medium; and
the storage of the plurality of input data sets is non-volatile type data storage.