Fetching query results through cloud object stores
The system is configured to: 1) receive a client request; 2) determine executor(s) to generate a response to the user request; 3) provide each of the executor(s) with an indication; 4) receive for each indication a response including an output of either a cloud output or an in-line output to generate a group of in-line outputs and a group of cloud outputs; 5) determine whether the group of in-line outputs comprises all outputs; and 6) in response to the group of in-line outputs not comprising all the outputs for the client request: a) convert the group of in-line outputs to a converted group of cloud outputs; b) generate metadata for the converted group of cloud outputs and the group of cloud outputs; and c) provide response to the client request including the metadata for the converted group of cloud outputs and the group of cloud outputs.
1. A method comprising:
receiving a request to read a set of query results stored in a cloud based data storage;
transmitting, to a client device, a first batch of the set of query results in response to the request, the first batch of the set of query results being a subset of the set of query results;
monitoring a time of transmission of the first batch of the set of query results;
comparing the time of transmission of the first batch of the set of query results to be received by the client device with transgresses a fast transfer threshold value;
responsive to the comparison indicating that the time of transmission of the first batch is less than the fast transfer threshold value, selecting a second batch of the set of query results, wherein a size of the second batch of the set of query results is greater than the size of the first batch;
transmitting the second batch of the set of query results to the client device;
monitoring the time of transmission of a third batch of the set of query results;
comparing the time of transmission of the third batch of the set of query results to be received by the client device with a slow transfer threshold value;
responsive to the comparison indicating that the time of transmission of the third batch is greater than the slow transfer threshold value, selecting a fourth batch of the set of query results, wherein a size of the fourth batch of the set of query results is less than the size of the third batch; and
transmitting the fourth batch of the set of query results to the client device.
2. The method of claim 1 , further comprising:
receiving, from the client device, a confirmation message indicating that the first batch of the set of query results has been received by the client device; and
determining the time of transmission of the first batch of the set of query results to be received by the client device based on the confirmation message.
3. The method of claim 1 , further comprising:
receiving a client request to generate a query;
generating, with a set of executors, the set of query results based on the query;
storing the set of query results in the cloud based data storage; and
returning metadata indicating a location of the set of query results in response to the client request.
4. The method of claim 3 , wherein the request to read the set of query results stored in a cloud based data storage includes the metadata indicating the location of the set of query results.
5. The method of claim 1 , wherein monitoring a time of transmission uses a timer to track the time elapsed between a generation and an attempt to access and download batches of files from a cloud storage system.
6. A cloud computation system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the cloud computation system to:
receive a request to read a set of query results stored in a cloud based data storage;
transmit, to a client device, a first batch of the set of query results in response to the request, the first batch of the set of query results being a subset of the set of query results;
monitor a time of transmission of the first batch of the set of query results;
compare the time of transmission the first batch of the set of query results to be received by the client device with a fast transfer threshold value;
responsive to the comparison indicating that the time of transmission of the first batch is less than the fast transfer threshold value, select a second batch of the set of query results, wherein a size of the second batch of the set of query results is greater than the size of the first batch;
transmit the second batch of the set of query results to the client device;
monitor the time of transmission of a third batch of the set of query results;
compare the time of transmission of the third batch of the set of query results to be received by the client device with a slow transfer threshold value;
responsive to the comparison indicating that the time of transmission of the third batch is greater than the slow transfer threshold value, select a fourth batch of the set of query results, wherein a size of the fourth batch of the set of query results is less than the size of the third batch; and
transmit the fourth batch of the set of query results to the client device.
7. The cloud computation system of claim 6 , wherein the instructions further cause the cloud computation system to:
receive, from the client device, a confirmation message indicating that the first batch of the set of query results has been received by the client device; and
determine the time of transmission of the first batch of the set of query results to be received by the client device based on the confirmation message.
8. The cloud computation system of claim 6 , wherein the instructions further cause the cloud computation system to:
receive a client request to generate a query;
generate, with a set of executors, the set of query results based on the query;
store the set of query results in the cloud based data storage; and
return metadata indicating a location of the set of query results in response to the client request.
9. The cloud computation system of claim 8 , wherein the request to read the set of query results stored in a cloud based data storage includes the metadata indicating the location of the set of query results.
10. The cloud computation system of claim 8 , wherein monitoring a time of transmission uses a timer to track the time elapsed between a generation and an attempt to access and download batches of files from a cloud storage system.
11. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a cloud computation system, cause the cloud computation system to perform operations comprising:
receive a request to read a set of query results stored in a cloud based data storage;
transmit, to a client device, a first batch of the set of query results in response to the request, the first batch of the set of query results being a subset of the set of query results;
monitor a time of transmission of the first batch of the set of query results;
compare the time of transmission of the first batch of the set of query results to be received by the client device with transgresses a fast transfer threshold value;
responsive to the comparison indicating that the time of transmission of the first batch is less than the fast transfer threshold value, select a second batch of the set of query results, wherein a size of the second batch of the set of query results is greater than the size of the first batch;
transmit the second batch of the set of query results to the client device;
monitor the time of transmission of a third batch of the set of query results;
compare the time of transmission of the third batch of the set of query results to be received by the client device with a slow transfer threshold value;
responsive to the comparison indicating that the time of transmission of the third batch is greater than the slow transfer threshold value, select a fourth batch of the set of query results, wherein a size of the fourth batch of the set of query results is less than the size of the third batch; and
transmit the fourth batch of the set of query results to the client device.
12. The non-transitory computer-readable medium of claim 11 , wherein the instructions further cause the cloud computation system to:
receive, from the client device, a confirmation message indicating that the first batch of the set of query results has been received by the client device; and
determine the time of transmission of the first batch of the set of query results to be received by the client device based on the confirmation message.
13. The non-transitory computer-readable medium of claim 11 , wherein the instructions further cause the cloud computation system to:
receive a client request to generate a query;
generate, with a set of executors, the set of query results based on the query;
store the set of query results in the cloud based data storage; and
return metadata indicating a location of the set of query results in response to the client request, wherein the request to read the set of query results stored in a cloud based data storage includes the metadata indicating the location of the set of query results.
14. The non-transitory computer-readable medium of claim 13 , wherein the request to read the set of query results stored in a cloud based data storage includes the metadata indicating the location of the set of query results.
15. The non-transitory computer-readable medium of claim 11 , wherein monitoring a time of transmission uses a timer to track the time elapsed between a generation and an attempt to access and download batches of files from a cloud storage system.