Time series data management systems and methods
This disclosure relates to, among other things, scalable data processing, storage, and/or management systems and methods. Certain embodiments disclosed herein provide for a multi-dimensional data storage structure that may facilitate parallel processing of data during both data ingestion and data retrieval and/or access processes involving one or more storage layers In various embodiments, sequence number schemes associated with ingested data records may allow for improved management of data stored in one or more layers (e.g., hot and/or cold data storage layers).
1. A computer-implemented method for managing time series data performed by a data management service system, the method comprising:
receiving a first time series data record included in a time series data stream, the first time series data record being associated with a first data topic and comprising a first timestamp;
assigning a first sequence number to the first time series data record;
determining, by a streaming writer service, based on information accessed from a definitions metastore and the first data topic, that the first time series data record is associated with a data topic indicated for storage in a hot data store;
writing, by the streaming writer service, the first time series data record to a first data partition included in the hot data store;
writing, by a canonical store writer service, the first time series data record to a canonical data store; and
writing, by a segment extraction service, the first time series data record from the canonical data store to a first data partition included in a cold data store based on the information accessed from the definitions metastore, the first data partition included in the cold data store being associated with a first time bucket, the first timestamp being associated with the first time bucket.
2. The computer-implemented method of claim 1 , wherein the first time series data record is associated with a topic offset number and the first sequence number comprises the first topic offset number.
3. The computer-implemented method of claim 1 , wherein the first sequence number is unique to the first data partition.
4. The computer-implemented method of claim 3 , wherein the first sequence number is greater than a sequence number assigned to a prior data record.
5. The computer-implemented method of claim 1 , wherein determining that the first time series data record is associated with a data topic indicated for storage in the hot data store comprises determining that the first data topic comprises a topic indicated in the information accessed from the definitions metastore for storage in the hot data store.
6. The computer-implemented method of claim 1 , wherein the first data partition is associated with the first data topic.
7. The computer-implemented method of claim 1 , wherein writing the first time series data record to the first data partition included in the hot data store further comprises repartitioning the time series data.
8. The computer-implemented method of claim 1 , wherein the hot data store comprises a cloud service data store.
9. The computer-implemented method of claim 1 , wherein the canonical data store comprises a cloud service data store.
10. The computer-implemented method of claim 1 , wherein the cold data store comprises a cloud service data store.
11. The computer-implemented method of claim 1 , wherein the data management service system is configured to execute the streaming writer service, the canonical store writer service, and the segment extraction service.
12. The computer-implemented method of claim 1 , wherein the method further comprises updating, in response to writing the first time series data record to the first data partition included in the hot data store, a shared data partitions index.
13. The computer-implemented method of claim 1 , wherein the method further comprises updating, in response to writing the first time series data record to the canonical data store, a canonical segment index.
14. The computer-implemented method of claim 1 , wherein the method further comprises updating, in response to writing the first time series data record from the canonical data store to the cold data store, a cold storage layer data segment index.
15. The computer-implemented method of claim 1 , wherein the method further comprises receiving a second time series data record, the second time series data record being associated with the first data topic and comprising the first timestamp.
16. The computer-implemented method of claim 15 , wherein the method further comprises assigning a second sequence number higher than the first sequence number to the second time series data record.
17. The computer-implemented method of claim 16 , wherein the method further comprises:
receiving, from a querying system, a request for access to a data record associated with the first timestamp;
determining that the second sequence number is higher than the first sequence number; and
providing access to the querying system of the second time series data record based on determining that the second sequence number is higher than the first sequence number.