Database system with run-time query mode selection

1. A computer-implemented method of processing database queries, the method comprising:

receiving, from a client device, a first query;

selecting a low-cost mode for processing the first query from a plurality of possible modes, the plurality of possible modes including a fast mode and the low-cost mode;

processing the first query according to the low-cost mode to generate first query results, wherein processing the first query according to the low-cost mode comprises:

querying a cloud database for index metadata of records that are responsive to the first query;

receiving, from the cloud database, the index metadata of records that are responsive to the first query; and

retrieving, using the index metadata, the records that are responsive to the first query from a datastore other than the cloud database; and

providing the first query results to the client device.

2. The computer-implemented method of claim 1 , further comprising:

receiving a second query;

selecting thefast mode to process the second query;

processing the second query according to the fast mode to generate second query results, wherein processing the second query according to the fast mode comprises:

forwarding the second query to a cloud database; and

receiving the second query results from the cloud database; and

providing the second query results in response to the second query.

3. The computer-implemented method of claim 2 , wherein the mode for processing the second query is selected by estimating a cost to process the second query using the fast mode and, responsive to the cost being below a cost threshold, selecting the fast mode to process the second query.

4. The computer-implemented method of claim 1 , wherein the index metadata for a responsive record includes a path and an offset, the path identifying a file in the datastore that includes the responsive record and the offset indicating a location of the responsive record in the file.

5. The computer-implemented method of claim 1 , wherein records including the responsive record were added to the datastore and the cloud database by a process comprising:

receiving data from one or more data sources;

generating the records based on the received data;

adding the records to a current record file;

adding the index metadata to the records;

saving the current record file to the datastore; and

loading the records in the current record file into a table in the cloud database.

6. The computer-implemented method of claim 5 , wherein the process for adding records to the datastore and the cloud database further comprises:

after adding the records to the current record file, determining whether one or more conditions are met; and

responsive to the one or more conditions being met, starting a new record file,

wherein the current record file is saved to the datastore and loaded into the table responsive to the one or more conditions being met.

7. The computer-implemented method of claim 1 , further comprising identifying a user that initiated the first query, wherein the mode for processing the first query is selected based on at least one of: a query mode selected by the user, a service tier of the user, or a query history of the user.

8. The computer-implemented method of claim 1 , further comprising identifying a user that initiated the first query, wherein the mode for processing the first query is selected using a process including:

estimating a cost to process the first query using the fast mode;

incrementing a tally of query costs incurred by the user in a given time period by the estimated cost; and

selecting the low-cost mode responsive to the incremented tally of query costs for the given time period exceeding a threshold.

9. Computer-implemented method of claim 1 , wherein the mode for processing the first query is selected by applying a machine-learning model to data associated with the first query.

10. The method of claim 1 , wherein the datastore comprises a permanent datastore and a cache, and retrieving the responsive records comprises:

responsive to determining a first record of the responsive records is available in the cache, retrieving the first record from the cache;

responsive to determining a second record of the responsive records is unavailable from the cache, retrieving the second record from the permanent datastore; and

storing the second record in the cache.

11. The method of claim 10 , wherein the second record is stored in the permanent datastore in a compressed format, the method further comprising:

decompressing the second record before storing the second record in the cache;

determining a percentage of the cache currently in use; and

responsive to the precentage of the cache exceeding a first threshold, recompressing the second record.

12. A non-transitory computer-readable storage medium storing instructions for processing database queries, the instructions, wherein executed by a computing system, causing the computing system to perform operations comprising:

receiving, from a client device, a first query;

selecting a low-cost mode for processing the first query from a plurality of possible modes, the plurality of possible modes including a fast mode and the low-cost mode;

processing the first query according to the low-cost mode to generate first query results, wherein processing the first query according to the low-cost mode comprises:

querying a cloud database for index metadata of records that are responsive to the first query;

receiving, from the cloud database, the index metadata of records that are responsive to the first query; and

retrieving, using the index metadata, the records that are responsive to the first query from a datastore other than the cloud database; and

providing the first query results to the client device.

13. The non-transitory computer-readable storage medium of claim 12 , wherein the operations further comprise:

receiving a second query;

selecting thefast mode to process the second query;

processing the second query according to the fast mode to generate second query results, wherein processing the second query according to the fast mode comprises:

forwarding the query to a cloud database; and

receiving the query results from the cloud database; and

providing the second query results in response to the second query.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the mode for processing the second query is selected by estimating a cost to process the second query using the fast mode and, responsive to the cost being below a cost threshold, selecting the fast mode to process the second query.

15. The non-transitory computer-readable storage medium of claim 12 , wherein the index metadata for a responsive record includes a path and an offset, the path identifying a file in the datastore that includes the responsive record and the offset indicating a location of the responsive record in the file.

16. The non-transitory computer-readable storage medium of claim 12 , wherein the operations further comprise identifying a user that initiated the query, and the mode for processing the first query is selected based on at least one of: a query mode selected by the user, a service tier of the user, or a query history of the user.

17. The non-transitory computer-readable storage medium of claim 12 , wherein the operations further comprise identifying a user that initiated the query, and the mode for processing the first query is selected using a process including:

estimating a cost to process the first query using the fast mode;

incrementing a tally of query costs incurred by the user in a given time period by the estimated cost; and

selecting the low-cost mode responsive to the incremented tally of query costs for the given time period exceeding a threshold.

18. The non-transitory computer-readable storage medium of claim 12 , wherein the mode for processing the first query is selected by applying a machine-learning model to data associated with the first query.

19. The non-transitory computer-readable storage medium of claim 12 , wherein the datastore comprises a permanent datastore and a cache, and retrieving the responsive records comprises:

responsive to determining a first record of the responsive records is available in the cache, retrieving the first record from the cache;

responsive to determining a second record of the responsive records is unavailable from the cache, retrieving the second record from the permanent datastore; and

storing the second record in the cache.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the second record is stored in the permanent datastore in a compressed format, the method further comprising:

decompressing the second record before storing the second record in the cache;

determining a percentage of the cache currently in use; and

responsive to the precentage of the cache exceeding a first threshold, recompressing the second record.