Analytic query processing using a backup of a database
Queries may be made against a non-relational database without impacting the capacities of the non-relational database by enabling performance of queries to a non-relational database via an external query management, backup management, and a non-relational database backup hosted in an external data storage. The query management and backup management of the non-relational database service may split transactional load to the non-relational database and analytical load, and furthermore allow selection of a desired data freshness for queries including various time ranges, such as seconds, minutes, or days.
1. A system, comprising:
one or more processors; and
a memory, storing program instructions that when executed by the one or more processors, cause the one or more processors to implement a non-relational database service, the non-relational database service configured to:
receive, via an interface of a non-relational database service that connects the non-relational database service with an external analytics service, a request to enable performance of queries by the external analytics service to the non-relational database hosted by the non-relational database service;
enable performance of the queries by the external analytics service to the non-relational database, comprising:
generate, by the non-relational database service, metadata for a backup of the non-relational database in an external object storage service, wherein the metadata describes:
the non-relational database system sorted according to one or more index keys and respective timestamps associated with write requests to the non-relational database; and
partition information that enables the external analytics service to query the backup of the non-relational database in the object storage service; and
making the metadata available to the external analytics service.
2. The system of claim 1 , wherein the non-relational database service is further configured to:
store logs associated with write requests to the non-relational database in the object storage service, wherein respective ones of the logs are associated with respective time ranges; and
perform horizontally distributed compaction of the logs into a single log associated with a single time range, wherein the backup of the non-relational database in the external data store is furthermore stored in a per-partition, log data structure able to perform horizontally distributed compaction.
3. The system of claim 2 , wherein the log data structure is a log-structured merge-tree.
4. The system of claim 1 , wherein the non-relational database service is further configured to:
store logs associated with write requests to the non-relational database in the object storage service, wherein respective ones of the logs are associated with respective time ranges;
perform horizontally distributed compaction of the logs and the backup of the non-relational database that is stored in the object storage service in a per-partition, log data structure able to perform horizontally distributed compaction, wherein the horizontally distributed compaction is performed upon determination that:
a compute and storage cost of the performance of the compaction is offset by cost savings for performance of one or more subsequent queries on the backup, or
a threshold target latency in performance of one or more subsequent queries on the backup would be breached without the performance of the compaction.
5. The system of claim 1 , wherein the non-relational database service is further configured to:
update the backup of the non-relational database based on a passage of a predetermined threshold of time regardless of whether a threshold volume of write requests has been met.
6. A method, comprising:
receiving, via an interface of a non-relational database system that connects the non-relational database system with an external query engine, a request to enable performance of queries by the external query engine to the non-relational database hosted by the non-relational database system; and
enabling performance of the queries by the external query engine to the non-relational database, comprising:
generating, by the non-relational database system, metadata for a backup of the non-relational database in an external data store, wherein the metadata describes:
backup of the non-relational database sorted according to one or more index keys and respective timestamps associated with write requests to the non-relational database;
partition information that enables the external query engine to query the backup of the non-relational database in the data store; and
making the metadata available to the external query engine.
7. The method of claim 6 , further comprising:
storing logs associated with write requests to the non-relational database in the external data store, wherein respective ones of the logs are associated with respective time ranges; and
performing horizontally distributed compaction of the logs into a single log associated with a single time range, wherein the backup of the non-relational database in the external data store is furthermore stored in a per-partition log data structure able to perform horizontally distributed compaction.
8. The method of claim 7 , wherein the log data structure is a log-structured merge-tree.
9. The method of claim 6 , further comprising:
storing logs associated with write requests to the non-relational database in the external data store, wherein respective ones of the logs are associated with respective time ranges;
performing horizontally distributed compaction of the logs and the backup of the non-relational database that is stored in the external data store in a per-partition log data structure able to perform horizontally distributed compaction, wherein the horizontally distributed compaction is performed upon determination that:
a compute and storage cost of the performance of the compaction is offset by cost savings for performance of one or more subsequent queries on the backup, or
a threshold target latency in performance of one or more subsequent queries on the backup would be breached without the performance of the compaction.
10. The method of claim 6 , further comprising:
updating the backup of the non-relational database based on a passage of a predetermined threshold of time regardless of whether a threshold volume of write requests has been met.
11. The method of claim 6 , further comprising:
generating the backup of the non-relational database in the external data store, wherein generating the backup comprises:
reading change logs stored in the non-relational database system;
sorting the change logs to be sorted according to the one or more index keys and the respective timestamps associated with write requests to the non-relational database; and
storing the sorted change logs and producing a new partition file based on the sorted change logs, wherein the metadata comprises the new partition file.
12. The method of claim 11 , further comprising:
reading the partition file, prior to the performing the compaction, to determine that a threshold non-relational database size is met; and
performing a horizontally distributed compaction of the copy according to the respective range of the one or more index keys upon the determination that the threshold non-relational database size is met.
13. The method of claim 11 , further comprising:
identifying one or more portions of the backup that are not referenced by the partition file within a first threshold of time; and
deleting, subsequent to the compacting, the partition file upon determination that the age of the partition file has passed a second threshold of time.
14. The method of claim 6 , wherein the non-relational database is in a row-oriented file format, and wherein the backup of the non-relational database is in a columnar file format.
15. One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement a non-relational database service that implements:
receiving, via an interface of a non-relational database system that connects the non-relational database system with an external query engine, a request to enable performance of queries by the external query engine to the non-relational database hosted by the non-relational database system; and
enabling performance of the queries by the external query engine to the non-relational database, comprising:
generating, by the non-relational database system, metadata for a backup of the non-relational database in an external data store, wherein the metadata describes:
backup of the non-relational database sorted according to one or more index keys and respective timestamps associated with write requests to the non-relational database;
partition information that enables the external query engine to query the backup of the non-relational database in the data store; and
making the metadata available to the external query engine.
16. The one or more non-transitory, computer-readable storage media of claim 15 , wherein the non-relational database service further implements:
storing logs associated with write requests to the non-relational database in the external data store, wherein respective ones of the logs are associated with respective time ranges; and
performing horizontally distributed compaction of the logs into a single log associated with a single time range, wherein the backup of the non-relational database in the external data store is furthermore stored in a per-partition log data structure able to perform horizontally distributed compaction.
17. The one or more non-transitory, computer-readable storage media of claim 16 , wherein the log data structure is a log-structured merge-tree.
18. The one or more non-transitory, computer-readable storage media of claim 15 , wherein the non-relational database service further implements:
storing logs associated with write requests to the non-relational database in the external data store, wherein respective ones of the logs are associated with respective time ranges;
performing horizontally distributed compaction of the logs and the backup of the non-relational database that is stored in the external data store in a per-partition, log data structure able to perform horizontally distributed compaction, wherein the horizontally distributed compaction is performed upon determination that:
a compute and storage cost of the performance of the compaction is offset by cost savings for performance of one or more subsequent queries on the backup, or
a threshold target latency in performance of one or more subsequent queries on the backup would be breached without the performance of the compaction.
19. The one or more non-transitory, computer-readable storage media of claim 15 , wherein the non-relational database service further implements:
updating the backup of the non-relational database based on a passage of a predetermined threshold of time regardless of whether a threshold volume of write requests has been met.
20. The one or more non-transitory, computer-readable storage media of claim 15 , wherein the non-relational database service further implements:
generating the backup of the non-relational database in the external data store, wherein generating the backup comprises:
reading change logs stored in the non-relational database system;
sorting the change logs to be sorted according to the one or more index keys and the respective timestamps associated with write requests to the non-relational database; and
storing the sorted change logs and producing a new partition file based on the sorted change logs, wherein the metadata comprises the new partition file.