CACHING SYSTEMS AND METHODS
Caching systems and methods are described. In one implementation, a method identifies multiple files used to process a query and distributes each of the multiple files to a particular execution node to execute the query. Each execution node determines whether the distributed file is stored in the execution node's cache. If the execution node determines that the file is stored in the cache, it processes the query using the cached file. If the file is not stored in the cache, the execution node retrieves the file from a remote storage device, stores the file in the execution node's cache, and processes the query using the file.
1 . A method comprising:
identifying, based on a received query, a plurality of tasks for processing the received query, wherein the query is directed to a plurality of files stored across a plurality of remote storage devices;
referencing a metadata store to determine whether a file associated with a task of the plurality of tasks is cached at least in part by one or more data nodes of an execution platform comprising a plurality of data nodes;
in response to determining that the file is cached at least in part by one or more data nodes, assigning processing of the task to a data node of the one or more data nodes that has cached at least a part of the file; and
in response to determining that the file is not cached entirely by the data node, retrieving, by a processor, one or more parts of the file that are not cached by the data node from one or more remote storage devices of the plurality of remote storage devices.
2 . The method of claim 1 , further comprising writing the one or more parts to a cache memory of the data node.
3 . The method of claim 1 , wherein each of the plurality of data nodes comprises a central processing unit (CPU) and a cache memory.
4 . The method of claim 3 , wherein the metadata store comprises metadata indicating an organization of the plurality of files across the plurality of remote storage devices and the plurality of cache memories.
5 . The method of claim 4 , further comprising:
updating the metadata of the metadata store to indicate that the file is completely stored in the cache memory of the data node.
6 . The method of claim 1 , wherein the plurality of files is stored across the plurality of remote storage devices in a columnar format, and parts of the file that are cached by the data node comprises columns of the file that are frequently accessed.
7 . The method of claim 1 , wherein the plurality of data nodes is organized into one or more clusters.
8 . The method of claim 1 , wherein the execution platform is separate and independent of the plurality of remote storage devices.
9 . The method of claim 8 , wherein at least the identifying, referencing, and assigning steps are performed by a resource manager that is separate from the execution platform.
10 . The method of claim 9 , wherein the resource manager is separate from the metadata store.
11 . A system comprising:
a memory; and
a processor, operatively coupled to the memory, the processor to:
identify, based on a received query, a plurality of tasks for processing the received query, wherein the query is directed to a plurality of files stored across a plurality of remote storage devices;
reference a metadata store to determine whether a file associated with a task of the plurality of tasks is cached at least in part by one or more data nodes of an execution platform comprising a plurality of data nodes;
in response to determining that the file is cached at least in part by one or more data nodes, assign processing of the task to a data node of the one or more data nodes that has cached at least a part of the file; and
in response to determining that the file is not cached entirely by the data node, retrieve one or more parts of the file that are not cached by the data node from one or more remote storage devices of the plurality of remote storage devices.
12 . The system of claim 11 , wherein the processor is further to write the one or more parts to a cache memory of the data node.
13 . The system of claim 11 , wherein each of the plurality of data nodes comprises central processing unit (CPU) and a cache memory.
14 . The system of claim 13 , wherein the metadata store comprises metadata indicating an organization of the plurality of files across the plurality of remote storage devices and the plurality of cache memories.
15 . The system of claim 14 , wherein the processor is further to:
update, by the data node, the metadata of the metadata store to indicate that the file is completely stored in the cache memory of the data node.
16 . The system of claim 11 , wherein the plurality of files is stored across the plurality of remote storage devices in a columnar format, and parts of the file that are cached by the data node comprise columns of the file that are frequently accessed.
17 . The system of claim 11 , wherein the plurality of data nodes is organized into one or more clusters.
18 . The system of claim 11 , wherein the execution platform is separate and independent of the plurality of remote storage devices.
19 . The system of claim 18 , wherein the processor performs at least the identifying, referencing, and assigning steps using a resource manager that is separate from the execution platform.
20 . The system of claim 19 , wherein the resource manager is separate from the metadata store.
21 . A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processor, cause the processor to:
identify, based on a received query, a plurality of tasks for processing the received query, wherein the query is directed to a plurality of files stored across a plurality of remote storage devices;
reference a metadata store to determine whether a file associated with a task of the plurality of tasks is cached at least in part by one or more data nodes of an execution platform comprising a plurality of data nodes;
in response to determining that the file is cached at least in part by one or more data nodes, assign processing of the task to a data node of the one or more data nodes that has cached at least a part of the file; and
in response to determining that the file is not cached entirely by the data node, retrieve, by the processor, one or more parts of the file that are not cached by the data node from one or more remote storage devices of the plurality of remote storage devices.
22 . The non-transitory computer-readable medium of claim 21 , wherein the processor is further to write the one or more parts to a cache memory of the data node.
23 . The non-transitory computer-readable medium of claim 21 , wherein each of the plurality of data nodes comprises central processing unit (CPU) and a cache memory.
24 . The non-transitory computer-readable medium of claim 23 , wherein the metadata store comprises metadata indicating an organization of the plurality of files across the plurality of remote storage devices and the plurality of cache memories.
25 . The non-transitory computer-readable medium of claim 24 , wherein the processor is further to:
update, by the data node, the metadata of the metadata store to indicate that the file is completely stored in the cache memory of the data node.
26 . The non-transitory computer-readable medium of claim 21 , wherein the plurality of files is stored across the plurality of remote storage devices in a columnar format, and parts of the file that are cached by the data node comprise columns of the file that are frequently accessed.
27 . The non-transitory computer-readable medium of claim 21 , wherein the plurality of data nodes is organized into one or more clusters.
28 . The non-transitory computer-readable medium of claim 21 , wherein the execution platform is separate and independent of the plurality of remote storage devices.
29 . The non-transitory computer-readable medium of claim 28 , wherein the processor performs at least the identifying, referencing, and assigning steps using a resource manager that is separate from the execution platform.
30 . The system of claim 19 , wherein the resource manager is separate from the metadata store.