Caching systems and methods
Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.
1 . A method comprising:
receiving, an execution platform of a data processing platform, a query directed to data stored across a plurality of shared storage devices in a multi-tenant database that isolates computing resources and data storage resources between different customers, wherein the execution platform comprises a plurality of virtual warehouses including a virtual warehouse;
selecting a computing system from amongst a plurality of computing systems to implement a first execution node in the virtual warehouse based on communication capabilities of networks within a geographic location, wherein the first execution node comprises a first cache, wherein the first cache stores at least a portion of the data, wherein each of the plurality of virtual warehouses is able to access all of a plurality of data storage resources of a storage platform that is independent from the execution platform, and wherein the virtual warehouse is able to access a data storage resource in the plurality of data storage resources at a same time as a second virtual warehouse in the plurality of virtual warehouses;
accessing, by a resource manager of the data processing platform and via a communication link implemented using a data communication network, a metadata database storing metadata that includes information regarding how the at least the portion of the data is organized in the first cache and in the storage platform, wherein the metadata is stored separately from the execution platform, and wherein the metadata identifies a subset of a plurality of rows of a table associated with the data and a subset of a plurality of columns associated with the data;
processing, based on the accessed metadata, the query using the first execution node of the execution platform to identify expected tasks of the first execution node related to the query;
creating a second execution node in the virtual warehouse based on the expected tasks of the first execution node related to the query, wherein the second execution node comprises a second cache, wherein a size of the first cache differs from a size of the second cache, wherein a speed of input-output operations of the first cache differs from a speed of input-output operations of the second cache; and
in response to creating the second execution node, reassigning, by a processing device, processing of at least a portion of the query to the second execution node based on at least one of the size of the first cache, the size of the second cache, the speed of the input-output operations of the first cache, or the speed of the input-output operations of the second cache.
2 . The method of claim 1 , wherein the multi-tenant database comprises a set of tables.
3 . The method of claim 2 , wherein at least one table of the set of tables is encrypted and is subsequently decrypted before executing the query.
4 . The method of claim 2 , wherein at least one table of the set of tables is compressed and is subsequently decompressed before executing the query.
5 . The method of claim 2 , wherein:
the first execution node comprises a plurality of processors and each processor of the plurality of processors processes one table of the set of tables; and
data from the processed one table of the set of tables is stored in a cache associated with a processor.
6 . The method of claim 1 , wherein the multi-tenant database is a relational database.
7 . The method of claim 6 , wherein the relational database is a structured query language database.
8 . The method of claim 1 , wherein the query is received from a client, the method further comprising:
generating a set of results; and
returning the set of results to the client.
9 . The method of claim 1 , further comprising:
optimizing the query.
10 . The method of claim 1 , wherein processing the query is based at least in part on a set of statistics.
11 . The method of claim 10 , wherein the set of statistics is automatically accumulated.
12 . The method of claim 10 , wherein the set of statistics is automatically updated.
13 . The method of claim 1 , wherein the metadata includes a summary of the at least the portion of the data.
14 . The method of claim 1 , wherein selecting the computing system from amongst the plurality of computing systems to implement the first execution node is additionally based on first communication capabilities of first networks between geographic locations.
15 . The method of claim 1 , wherein selecting the computing system from amongst the plurality of computing systems to implement the first execution node is additionally based on which computing systems in the plurality of computing systems are currently implementing other execution nodes.
16 . A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
receive, at an execution platform of a data processing platform, a query directed to data stored across a plurality of shared storage devices in a multi-tenant database that isolates computing resources and data storage resources between different customers, wherein the execution platform comprises a plurality of virtual warehouses including a virtual warehouse;
select a computing system from amongst a plurality of computing systems to implement a first execution node in the virtual warehouse based on communication capabilities of networks within a geographic location, wherein the first execution node comprises a first cache, wherein the first cache stores at least a portion of the data, wherein each of the plurality of virtual warehouses is able to access all of a plurality of data storage resources of a storage platform that is independent from the execution platform, and wherein the virtual warehouse is able to access a data storage resource in the plurality of data storage resources at a same time as a second virtual warehouse in the plurality of virtual warehouses;
access, by a resource manager of the data processing platform and via a communication link implemented using a data communication network, a metadata database storing metadata that includes information regarding how the at least the portion of the data is organized in the first cache and in the storage platform, wherein the metadata is stored separately from the execution platform, and wherein the metadata identifies a subset of a plurality of rows of a table associated with the data and a subset of a plurality of columns associated with the data;
process, based on the accessed metadata, the query using the first execution node of the execution platform to identify expected tasks of the first execution node related to the query;
create a second execution node in the virtual warehouse based on the expected tasks of the first execution node related to the query, wherein the second execution node comprises a second cache, wherein a size of the first cache differs from a size of the second cache, wherein a speed of input-output operations of the first cache differs from a speed of input-output operations of the second cache; and
in response to the creation of the second execution node, reassign processing of at least a portion of the query to the second execution node based on at least one of the size of the first cache, the size of the second cache, the speed of the input-output operations of the first cache, or the speed of the input-output operations of the second cache.
17 . The system of claim 16 , wherein the first cache includes a memory device and a disk storage device.
18 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
receive, at an execution platform of a data processing platform, a query directed to data stored across a plurality of shared storage devices in a multi-tenant database that isolates computing resources and data storage resources between different customers, wherein the execution platform comprises a plurality of virtual warehouses including a virtual warehouse;
select a computing system from amongst a plurality of computing systems to implement a first execution node in the virtual warehouse based on communication capabilities of networks within a geographic location, wherein the first execution node comprises a first cache, wherein the first cache stores at least a portion of the data, wherein each of the plurality of virtual warehouses is able to access all of a plurality of data storage resources of a storage platform that is independent from the execution platform, and wherein the virtual warehouse is able to access a data storage resource in the plurality of data storage resources at a same time as a second virtual warehouse in the plurality of virtual warehouses;
access, by a resource manager of the data processing platform and via a communication link implemented using a data communication network, a metadata database storing metadata that includes information regarding how the at least the portion of the data is organized in the first cache and in the storage platform, wherein the metadata is stored separately from the execution platform, and wherein the metadata identifies a subset of a plurality of rows of a table associated with the data and a subset of a plurality of columns associated with the data;
process, based on the accessed metadata, the query using the first execution node of the execution platform to identify expected tasks of the first execution node related to the query;
creating a second execution node in the virtual warehouse based on the expected tasks of the first execution node related to the query, wherein the second execution node comprises a second cache, wherein a size of the first cache differs from a size of the second cache, wherein a speed of input-output operations of the first cache differs from a speed of input-output operations of the second cache; and
in response to the creation of the second execution node, reassign, by the processing device, processing of at least a portion of the query to the second execution node based on at least one of the size of the first cache, the size of the second cache, the speed of the input-output operations of the first cache, or the speed of the input-output operations of the second cache.
19 . The non-transitory computer-readable medium of claim 18 , wherein the query comprises a set of tables.
20 . The non-transitory computer-readable medium of claim 19 , wherein at least one table of the set of tables is encrypted and is subsequently decrypted before executing the query.
21 . The non-transitory computer-readable medium of claim 19 , wherein at least one table of the set of tables is compressed and is subsequently decompressed before executing the query.
22 . The non-transitory computer-readable medium of claim 19 , wherein:
the first execution node comprises a plurality of processors and each of the plurality of processors processes one table of the set of tables; and
data from the set of tables is stored in a cache associated with a processor.
23 . The non-transitory computer-readable medium of claim 18 , wherein the query is received from a client, and the instructions further cause the processing device to:
generate a set of results; and
return the set of results to the client.
24 . The non-transitory computer-readable medium of claim 18 , wherein the instructions further cause the processing device to:
optimize the query.
25 . The non-transitory computer-readable medium of claim 18 , wherein the multi-tenant database is a relational database.
26 . The non-transitory computer-readable medium of claim 25 , wherein the relational database is a structured query language database.
27 . The non-transitory computer-readable medium of claim 25 , wherein to reassign the processing of the at least the portion of the query, the instructions cause the processing device to reassign the processing of the at least the portion of the query based at least in part on a set of statistics.
28 . The non-transitory computer-readable medium of claim 27 , wherein the instructions further cause the processing device to:
accumulate the set of statistics automatically.
29 . The non-transitory computer-readable medium of claim 27 , wherein the instructions further cause the processing device to:
update the set of statistics automatically.
30 . The non-transitory computer-readable medium of claim 18 , wherein the metadata includes a summary of the at least the portion of the data.