Future scheduler for database systems

1. A system comprising:

at least one hardware processor; and

a memory storing instructions that cause the at least one hardware processor to perform operations comprising:

reserving, by a slot scheduler component, capacity to execute an incoming query via communication with a warehouse scheduling component, the warehouse scheduling component managing the capacity of a set of virtual warehouses for executing the incoming query, the reserved capacity comprising a set of slots, each slot comprising an allocation of resources provided by a particular virtual warehouse from the set of virtual warehouses, the slot scheduler component being provided by a compute service manager, the compute service manager being separate from the warehouse scheduling component, the communication with the warehouse scheduling component comprising the slot scheduler component sending a request for the capacity to execute the incoming query;

storing, by the warehouse scheduling component, information related to the request from the slot scheduler component, the information comprising a set of slots utilized by the slot scheduler component, a lease time per slot, and a timestamp of a last request received from the slot scheduler component;

receiving a query from a client device, the query comprising a set of query statements;

determining, by a query compiler during compilation of the query received from the client device, that the query, when executed, includes a runtime of less than a predetermined time, the query compiler being provided by the compute service manager, the compute service manager being separate from the set of virtual warehouses;

determining, by the slot scheduler component, that a set of resources to be utilized by the query during execution is provided by a slot from the set of slots, the slot corresponding to a specific amount of memory on a virtual warehouse for an amount of time, the determining based at least in part on determining that execution of the query does not consume more than the specific amount of memory as provided by the slot;

performing a first scheduling process for execution of the query using the slot;

determining that an available slot is provided by the virtual warehouse; and

executing, using the virtual warehouse, the query using the available slot.

2. The system of claim 1 , wherein the operations further comprise:

determining that the set of resources to be utilized by the query is not provided by the slot; and

performing a second scheduling process without using the slot.

3. The system of claim 2 , wherein performing the second scheduling process without using the slot comprises:

sending the query to a warehouse scheduling component to perform the second scheduling process.

4. The system of claim 1 , wherein determining that the set of resources to be utilized by the query comprises:

determining that the query is considered a lightweight query based on an analysis performed by the query compiler during compilation of the query.

5. The system of claim 4 , wherein the lightweight query comprises a runtime of less than a particular period of time, a degree of parallelism of one, and a memory consumption of less than a threshold percentage of a total available memory of the virtual warehouse.

6. The system of claim 1 , wherein performing the first scheduling process for execution of the query using the slot comprises:

sending a request for at least one slot from a warehouse scheduling component.

7. The system of claim 1 , wherein determining that the available slot is provided by the virtual warehouse comprises:

sending a request to renew a lease of the available slot.

8. The system of claim 1 , wherein executing, using the virtual warehouse, the query using the available slot comprises:

sending the query to the virtual warehouse for execution.

9. The system of claim 1 , wherein the operations further comprise:

requesting a new slot corresponding to the available slot; and

waiting for the available slot by placing the query in a queue for the new slot.

10. The system of claim 1 , wherein the operations further comprise:

determining, by the query compiler, that the query, when executed, utilizes less than a particular amount of memory; and

in response to determining that the query, when executed, includes the runtime of less than the predetermined time and that the query utilizes less than the particular amount of memory, indicating, by the query compiler, that the query qualifies as a lightweight query.

11. The system of claim 10 , wherein the operations further comprise:

prior to determining, by the slot scheduler component, that the set of resources to be utilized by the query during execution is provided by the slot:

in response to the query qualifying as the lightweight query based at least in part on the indicating, sending the query to the slot scheduler component to select an available slot for executing the query.

12. The system of claim 11 , wherein the operations further comprise:

determining, by the slot scheduler component, that no slot from the set of slots is available;

placing, by the slot scheduler component, the query in a queue in response to determining that no slot is available; and

sending a notification, indicating that no slot is available, to the warehouse scheduling component to allocate an additional set of slots.

13. A method comprising:

reserving, by a slot scheduler component, capacity to execute an incoming query via communication with a warehouse scheduling component, the warehouse scheduling component managing the capacity of a set of virtual warehouses for executing the incoming query, the reserved capacity comprising a set of slots, each slot comprising an allocation of resources provided by a particular virtual warehouse from the set of virtual warehouses, the slot scheduler component being provided by a compute service manager, the compute service manager being separate from the warehouse scheduling component, the communication with the warehouse scheduling component comprising the slot scheduler component sending a request for the capacity to execute the incoming query;

storing, by the warehouse scheduling component, information related to the request from the slot scheduler component, the information comprising a set of slots utilized by the slot scheduler component, a lease time per slot, and a timestamp of a last request received from the slot scheduler component;

receiving a query from a client device, the query comprising a set of query statements;

determining, by a query compiler during compilation of the query received from the client device, that the query, when executed, includes a runtime of less than a predetermined time, the query compiler being provided by the compute service manager, the compute service manager being separate from the set of virtual warehouses;

determining, by a scheduler component, that a set of resources to be utilized by the query during execution is provided by a slot from the set of slots, the slot corresponding to a specific amount of memory on a virtual warehouse for an amount of time, the determining based at least in part on determining that execution of the query does not consume more than the specific amount of memory as provided by the slot;

performing a first scheduling process for execution of the query using the slot;

determining that an available slot is provided by the virtual warehouse; and

executing, using the virtual warehouse, the query using the available slot.

14. The method of claim 13 , further comprising:

determining that the set of resources to be utilized by the query is not provided by the slot; and

performing a second scheduling process without using the slot.

15. The method of claim 14 , wherein performing the second scheduling process without using the slot comprises:

sending the query to a warehouse scheduling component to perform the second scheduling process.

16. The method of claim 13 , wherein determining that the set of resources to be utilized by the query comprises:

determining that the query is considered a lightweight query based on an analysis performed by the query compiler during compilation of the query.

17. The method of claim 16 , wherein the lightweight query comprises a runtime of less than a particular period of time, a degree of parallelism of one, and a memory consumption of less than a threshold percentage of a total available memory of the virtual warehouse.

18. The method of claim 13 , wherein performing the first scheduling process for execution of the query using the slot comprises:

sending a request for at least one slot from a warehouse scheduling component.

19. The method of claim 13 , wherein determining that the available slot is provided by the virtual warehouse comprises:

sending a request to renew a lease of the available slot.

20. The method of claim 13 , wherein executing, using the virtual warehouse, the query using the available slot comprises:

sending the query to the virtual warehouse for execution.

21. The method of claim 13 , further comprising:

requesting a new slot corresponding to the available slot; and

waiting for the available slot by placing the query in a queue for the new slot.

22. A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

reserving, by a slot scheduler component, capacity to execute an incoming query via communication with a warehouse scheduling component, the warehouse scheduling component managing the capacity of a set of virtual warehouses for executing the incoming query, the reserved capacity comprising a set of slots, each slot comprising an allocation of resources provided by a particular virtual warehouse from the set of virtual warehouses, the slot scheduler component being provided by a compute service manager, the compute service manager being separate from the warehouse scheduling component, the communication with the warehouse scheduling component comprising the slot scheduler component sending a request for the capacity to execute the incoming query;

storing, by the warehouse scheduling component, information related to the request from the slot scheduler component, the information comprising a set of slots utilized by the slot scheduler component, a lease time per slot, and a timestamp of a last request received from the slot scheduler component;

receiving a query from a client device, the query comprising a set of query statements;

determining, by a query compiler during compilation of the query received from the client device, that the query, when executed, includes a runtime of less than a predetermined time, the query compiler being provided by the compute service manager, the compute service manager being separate from the set of virtual warehouses;

determining, by the slot scheduler component, that a set of resources to be utilized by the query during execution is provided by a slot from the set of slots, the slot corresponding to a specific amount of memory on a virtual warehouse for an amount of time, the determining based at least in part on determining that execution of the query does not consume more than the specific amount of memory as provided by the slot;

performing a first scheduling process for execution of the query using the slot;

determining that an available slot is provided by the virtual warehouse; and

executing, using the virtual warehouse, the query using the available slot.

23. The non-transitory computer-storage medium of claim 22 , wherein the operations further comprise:

determining that the set of resources to be utilized by the query is not provided by the slot; and

performing a second scheduling process without using the slot.

24. The non-transitory computer-storage medium of claim 23 , wherein performing the second scheduling process without using the slot comprises:

sending the query to a warehouse scheduling component to perform the second scheduling process.

25. The non-transitory computer-storage medium of claim 22 , wherein determining that the set of resources to be utilized by the query comprises:

determining that the query is considered a lightweight query based on an analysis performed by the query compiler during compilation of the query.

26. The non-transitory computer-storage medium of claim 25 , wherein the lightweight query comprises a runtime of less than a particular period of time, a degree of parallelism of one, and a memory consumption of less than a threshold percentage of a total available memory of the virtual warehouse.

27. The non-transitory computer-storage medium of claim 22 , wherein performing the first scheduling process for execution of the query using the slot comprises:

sending a request for at least one slot from a warehouse scheduling component.

28. The non-transitory computer-storage medium of claim 22 , wherein determining that the available slot is provided by the virtual warehouse comprises:

sending a request to renew a lease of the available slot.

29. The non-transitory computer-storage medium of claim 22 , wherein executing, using the virtual warehouse, the query using the available slot comprises:

sending the query to the virtual warehouse for execution.

30. The non-transitory computer-storage medium of claim 22 , wherein the operations further comprise:

requesting a new slot corresponding to the available slot; and

waiting for the available slot by placing the query in a queue for the new slot.