Content-aware storage tiering techniques within a job scheduling system
View Patent ↗Techniques are provided for content-aware storage tiering techniques within a job scheduling system. A data node in a job scheduling environment receives at least one job to be executed for a portion of a larger logical file, and notifies one or more sibling data nodes to prefetch one or more related portions of the larger logical file. A write operation performed by an application client to store data on the data node is optionally intercepted at the data node and the associated data is stored in a performance storage tier of the data node. The data is optionally moved from the performance storage tier to a capacity storage tier if one or more predefined eviction criteria are satisfied. Data objects to be evicted are selected based on an anticipated future access of the data objects. The eviction selection is optionally based on grey box knowledge of the application client and/or information obtained from the application client.
1. A method performed in a data node of a job scheduling environment, comprising:
receiving, by said data node, a read request for at least one job to be executed for a portion of a larger logical file; and
notifying, by said data node in response to said read request for said portion of said larger logical file, at least a second sibling data node in the job scheduling environment to prefetch one or more related portions of the larger logical file needed for one or more tasks to be executed by said at least second sibling data node, wherein said prefetch occurs before said one or more tasks are assigned to said at least second sibling data node, wherein said larger logical file is stored on one or more tiers of a multi-tier storage system using a single namespace across said multi-tier storage system for said data node and said one or more sibling data nodes.
2. The method of claim 1 further comprising the steps of intercepting at least one write operation perfoiined by an application client to store data on the data node and storing the data associated with the intercepted at least one write operation in a performance storage tier of the data node that employs storage resources based on performance considerations.
3. The method of claim 2 further comprising the step of moving the data from the performance storage tier if one or more predefined eviction criteria are satisfied to a capacity storage tier that employs storage resources based on capacity considerations.
4. The method of claim 3 wherein the one or more predefined eviction criteria comprises an additional capacity needed in the performance storage tier.
5. The method of claim 1 further comprising the step of evicting at least one of the portions of the larger logical file based on an anticipated future access of the at least one portion.
6. The method of claim 5 wherein the anticipated future access is based on one or more of grey box knowledge of the application client and information obtained from the application client.
7. The method of claim 1 further comprising the step of selecting one or more data objects to be evicted from one or more storage devices based on an anticipated future access of the one or more data objects.
8. The method of claim 7 wherein the selection of the one or more data objects for eviction is based on one or more of grey box knowledge of the application client and information obtained from the application client.
9. The method of claim 1 wherein a plurality of the data nodes share a flash storage system and wherein one of the data nodes prefetches the one or more related portions of the larger logical file on behalf of the one or more sibling data nodes.
10. A computer program product comprising a non-transitory processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed perform the steps of the method of claim 1 .
11. A data node apparatus of a job scheduling environment, comprising:
a memory; and
at least one hardware device, coupled to the memory, operative to implement the following steps:
receiving, by said data node apparatus, a read request for at least one job to be executed for a portion of a larger logical file; and
notifying, by said data node apparatus in response to said read request for said portion of said larger logical file, at least a second sibling data node in the job scheduling environment to prefetch one or more related portions of the larger logical file needed for one or more tasks to be executed by said at least second sibling data node, wherein said prefetch occurs before said one or more tasks are assigned to said at least second sibling data node, wherein said larger logical file is stored on one or more tiers of a multi-tier storage system using a single namespace across said multi-tier storage system for said data node and said one or more sibling data nodes.
12. The data node apparatus of claim 11 wherein the at least one hardware device is further configured to intercept at least one write operation performed by an application client to store data on the data node, store the data associated with the intercepted at least one write operation in a performance storage tier of the data node that employs storage resources based on performance considerations and move the data from the perfoiiiiance storage tier if one or more predefined eviction criteria are satisfied to a capacity storage tier that employs storage resources based on capacity considerations.
13. The data node apparatus of claim 11 wherein the at least one hardware device is further configured to select one or more data objects to be evicted from one or more storage devices based on an anticipated future access of the one or more data objects.
14. The data node apparatus of claim 13 wherein the selection of the one or more data objects for eviction is based on one or more of grey box knowledge of the application client and information obtained from the application client.
15. The data node apparatus of claim 11 wherein the capacity storage tier comprises a shared capacity object tier.
16. The data node apparatus of claim 11 wherein a plurality of the data nodes share a flash storage system and wherein one of the data nodes prefetches the one or more related portions of the larger logical file on behalf of the one or more sibling data nodes.
17. A job scheduling system comprising:
a job scheduling node; and
a plurality of data nodes, wherein at least one of the data nodes is configured to:
receive, by a first data node, a read request for at least one job to be executed for a portion of a larger logical file; and
notify, by said first data node in response to said read request for said portion of said larger logical file, at least a second sibling data node in the job scheduling environment to prefetch one or more related portions of the larger logical file needed for one or more tasks to be executed by said at least second sibling data node, wherein said prefetch occurs before said one or more tasks are assigned to said at least second sibling data node, wherein said larger logical file is stored on one or more tiers of a multi-tier storage system using a single namespace across said multi-tier storage system for said data node and said one or more sibling data nodes.
18. The job scheduling system of claim 17 wherein the at least one hardware device is further configured to intercept at least one write operation performed by an application client to store data on the data node, store the data associated with the intercepted at least one write operation in a performance storage tier of the data node that employs storage resources based on performance considerations and move the data from the performance storage tier if one or more predefined eviction criteria are satisfied to a capacity storage tier that employs storage resources based on capacity considerations.
19. The job scheduling system of claim 17 wherein the at least one hardware device is further configured to select one or more data objects to be evicted from one or more storage devices based on an anticipated future access of the one or more data objects.
20. The job scheduling system of claim 19 wherein the selection of the one or more data objects for eviction is based on one or more of grey box knowledge of the application client and information obtained from the application client.
21. The method of claim 1 , wherein a daemon executing on said data node queries an interface of a name node to determine a locality of said one or more related portions of the larger logical file and sends a network message to the one or more sibling data nodes that host the one or more related portions to notify the one or more sibling data nodes to prefetch said one or more related portions into respective performance tiers.