Prefetch chaining for optimizing data read-ahead for workflow and analytics

1. A method for optimizing data read-ahead for analytical workflows and analytics applications comprising:

obtaining, by a processor, next file information from a workflow scheduler for next files for a next processing stage that are to be accessed by a process;

prefetching data for the next processing stage for the process and at least one system job;

prefetching the next files as the prefetching data reaches an end of current inputs; and

causing a schedule to chain the next files for the next processing stage to current files of a current processing stage;

wherein the workflow scheduler reuses a computed prefetch distance as an initial prefetch distance for the next processing stage, and subsequent prefetch distances are continuously monitored and adjusted based on a trade-off determination that is determined between buffer space used and observed input/output (I/O) latency.

2. The method of claim 1 , wherein the process comprises an analytical workflow or an analytics application, prefetching data is performed continuously across files without waiting for a first user access to a file, a function provides for the schedule to form a chain that includes a predefined sequence for accessing a set of files that includes the next files and the current files, and the initial prefetch distance is an estimated prefetch distance.

3. The method of claim 2 , wherein the next files are next input files, the current files are current input files, and the trade-off determination is determined between buffer space used and observed input/output (I/O) latency.

4. The method of claim 2 , wherein a read-ahead cache manager prefetches multiple next files for the next processing stage and preloads a next stage executable.

5. The method of claim 2 , wherein the analytical workflow or analytics application comprises a model application comprising at least one machine learning model or a deep learning model, and the model application continues prefetching to a start of a same training file.

6. The method of claim 5 , wherein:

prefetching data is performed continuously across the model application and the at least one system job working sets;

prefetching is performed across file boundaries; and

native file system prefetching is integrated with a job scheduler.

7. The method of claim 6 , wherein the analytical workflow or analytics application provides a file system with a data flow or a stage sequence for prefetching a working set of data for the analytical workflow or analytics application using a read-ahead function for the file system and a read-ahead extension for the scheduler.

8. A computer program product for optimizing data read-ahead for analytical workflows and analytics applications, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

obtain, by the processor, next file information from a workflow scheduler for next files for a next processing stage that are to be accessed by a process;

prefetch, by the processor, data for the next processing stage for the process and at least one system job; prefetch, by the processor, the next files as the prefetching data reaches an end of current inputs; and

cause, by the processor, a schedule to chain the next files for the next processing stage to current files of a current processing stage, wherein the workflow scheduler reuses a computed prefetch distance as an initial prefetch distance for the next processing stage, and subsequent prefetch distances are continuously monitored and adjusted based on a trade-off determination that is determined between buffer space used and observed input/output (I/O) latency.

9. The computer program product of claim 8 , wherein the process comprises an analytical workflow or an analytics application, prefetching data is performed continuously across files without waiting for a first user access to a file, a function provides for the schedule to form a chain that includes a predefined sequence for accessing a set of files that includes the next files and the current files, and the initial prefetch distance is an estimated prefetch distance.

10. The computer program product of claim 9 , wherein the next files are next input files, the current files are current input files, and the trade-off determination is determined between buffer space used and observed input/output (I/O) latency.

11. The computer program product of claim 9 , wherein a read-ahead cache manager prefetches multiple next files for the next processing stage and preloads a next stage executable.

12. The computer program product of claim 9 , wherein the analytical workflow or analytics application comprises a model application comprising at least one machine learning model or a deep learning model, and the model application continues prefetching to a start of a same training file.

13. The computer program product of claim 2 , wherein:

prefetching data is performed continuously across the model application and the at least one system job working sets;

prefetching is performed across file boundaries; and

native file system prefetching is integrated with a job scheduler.

14. The computer program product of claim 13 , wherein the analytical workflow or analytics application provides a file system with a data flow or a stage sequence for prefetching a working set of data for the analytical workflow or analytics application using a read-ahead function for the file system and a read-ahead extension for the scheduler.

15. An apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to:

obtain next file information from a workflow scheduler for next files for a next processing stage that are to be accessed by a process;

prefetch data for the next processing stage for the process and at least one system job;

prefetch the next files as the prefetching data reaches an end of current inputs; and

cause a schedule to chain the next files for the next processing stage to current files of a current processing stage, wherein the workflow scheduler reuses a computed prefetch distance as an initial prefetch distance for the next processing stage, and subsequent prefetch distances are continuously monitored and adjusted based on a trade-off determination that is determined between buffer space used and observed input/output (I/O) latency.

16. The apparatus of claim 15 ; wherein:

the process comprises an analytical workflow or an analytics application;

prefetching data is performed continuously across files without waiting for a first user access to a file;

a function provides for the schedule to form a chain that includes a predefined sequence for accessing a set of files that includes the next files and the current files; and

the initial prefetch distance is an estimated prefetch distance.

17. The apparatus of claim 16 , wherein a read-ahead cache manager prefetches multiple next files for the next processing stage and preloads a next stage executable, the next files are next input files, the current files are current input files, and the trade-off determination is determined between buffer space used and observed input/output (I/O) latency.

18. The apparatus of claim 16 , wherein the analytical workflow or analytics application comprises a model application comprising at least one machine learning model or a deep learning model, and the model application continues prefetching to a start of a same training file.

19. The apparatus of claim 18 , wherein:

prefetching data is performed continuously across the model application and the at least one system job working sets;

prefetching is performed across file boundaries; and

native file system prefetching is integrated with a job scheduler.

20. The apparatus of claim 19 , wherein the analytical workflow or analytics application provides a file system with a data flow or a stage sequence for prefetching a working set of the workflow and analytics application using a read-ahead function for the file system and read-ahead extension for the scheduler.