System and method for file caching in a distributed program build environment
View Patent ↗A method is described comprising: scheduling jobs for a program build to execute in parallel across a plurality of nodes; predicting the files required to complete each of the jobs; and preloading the files to each node prior to the execution of each job.
1. A method comprising:
scheduling jobs for a program build to execute in parallel across a plurality of nodes;
predicting the files required to complete each of the jobs;
preloading the files to each node prior to the execution of each job; and
caching the files at the nodes across multiple jobs according to a specified cache management policy, the cache management policy comprises caching files of a first type on the nodes across multiple jobs and flushing files of a second type from the nodes after each job, wherein the first type of file is one which is likely to be used by multiple jobs whereas the second type of file is one which is unlikely to be used by multiple jobs and further wherein the first type of file is a header file and the second type of file is an object file.
2. The method as in claim 1 further comprising:
analyzing file dependency information to predict which files are required to complete each of the jobs.
3. The method as in claim 2 wherein the dependency information is contained within a Makefile.
4. The method as in claim 2 wherein the dependency information comprises file usage data collected from a previous program build.
5. The system as in claim 2 wherein the dependency information comprises both a Makefile and file usage data collected from a previous build.
6. The method as in claim 1 further comprising:
collecting file usage information from each job, the file usage information indicating the manner in which files are used at each job during the course of the program build;
storing the file usage information;
predicting the files required to complete each of the jobs in a subsequent build based on the file usage information; and
preloading the files to each node based on the predictions prior to the execution of each job.
7. The method as in claim 1 wherein the cache management policy divides files into a “shared” group which are to be cached on the nodes across multiple jobs and a “volatile” group which are to be flushed from the nodes after each job.
8. The method as in claim 7 further comprising:
initially tagging a plurality of files as volatile;
moving a first file from the volatile group to a shared group upon detecting that the first file is used in more than one job, wherein volatile files are flushed from node caches after each job and shared files are stored in node caches across multiple jobs.
9. The method as in claim 8 further comprising:
moving the first file back to the volatile group upon detecting that the first file has been modified during the program build.
10. The method as in claim 9 further comprising:
flushing the shared files from each node upon detecting that the first file has been modified.
11. A system comprising:
a job scheduler module to schedule jobs for a program build to execute in parallel across a plurality of nodes; and
a preload module to predict the files required to complete each of the jobs and pre-load the files to each node prior to the execution of each job; and
a caching module to cache the files at the nodes across multiple jobs according to a specified cache management policy, the cache management policy comprising caching files of a first type on the nodes across multiple jobs and flushing files of a second type from the nodes after each job, wherein the first type of file is one which is likely to be used by multiple jobs whereas the second type of file is one which is unlikely to be used by multiple jobs, further wherein a file of the first type is a header file and a file of the second type is an object file.
12. The system as in claim 11 wherein the preload module analyzes file dependency information to predict which files are required to complete each of the jobs.
13. The system as in claim 12 wherein the dependency information is contained within a Makefile.
14. The system as in claim 12 wherein the dependency information comprises file usage data collected from a previous program build.
15. The system as in claim 12 wherein the dependency information comprises both a Makefile and file usage data collected from a previous build.
16. The system as in claim 11 further comprising:
a usage tracking module to collect file usage information from each job, the file usage information indicating the manner in which files are used at each job during the course of the program build;
the preload module to predict the files required to complete each of the jobs in a subsequent build based on the file usage information, and preload the files to each node prior to the execution of each job in the subsequent build.
17. The system as in claim 11 wherein the cache management policy specifies dividing files into a “shared” group that is to be cached on the nodes across multiple jobs and a “volatile” group that is to be flushed from the nodes after each job.
18. The system as in claim 17 wherein the caching module initially tags a plurality of files as volatile and moves a first file from the volatile group to a shared group upon detecting that the first file is used in more than one job.
19. The system as in claim 18 wherein the caching module moves the first file back to the volatile group upon detecting that the first file has been modified during the program build.
20. The system as in claim 19 wherein the caching module flushes the shared files from each node upon detecting that the first file has been modified.
21. A method comprising:
scheduling jobs for a program build to execute in parallel across a plurality of nodes; and
caching a subset of the files at the nodes across multiple jobs, the subset of files selected for caching based on a specified cache management policy, the cache management policy comprising caching files of a first type on the nodes across multiple jobs and flushing files of a second type from the nodes after each job, wherein the first type of file is one that is likely to be used by multiple jobs whereas the second type of file is one that is unlikely to be used by multiple jobs, further wherein a file of the first type is a header file and a file of the second type is an object file.
22. A method comprising:
scheduling jobs for a program build to execute in parallel across a plurality of nodes; and
caching a subset of the files at the nodes across multiple jobs, the subset of files selected for caching based on a specified cache management policy, the cache management policy specifying dividing files into a shared group that are to be cached on the nodes across multiple jobs and a volatile group that is to be flushed from the nodes after each job.
23. The method as in claim 22 further comprising:
initially tagging a plurality of files as volatile;
moving a first file from the volatile group to a shared group upon detecting that the first file is used in more than one job, wherein volatile files are flushed from node caches after each job and shared files are stored in node caches across multiple jobs.
24. The method as in claim 23 further comprising:
moving the first file back to the volatile group upon detecting that the first file has been modified during the program build.
25. The method as in claim 24 further comprising:
flushing the shared files from each node upon detecting that the first file has been modified.
26. A method comprising:
scheduling jobs for a program build to execute in parallel across a plurality of nodes;
transmitting the files required to complete each of the jobs to each of the nodes; and
caching certain files at one or more of the nodes across multiple jobs according to a specified cache management policy, the cache management policy comprising caching files of a first type on the nodes across multiple jobs and flushing files of a second type from the nodes after each job, wherein the first type of file is one which is likely to be used by multiple jobs whereas the second type of file is one which is unlikely to be used by multiple jobs, further wherein a file of the first type is a header file and a file of the second type is an object file.
27. A method comprising:
scheduling jobs for a program build to execute in parallel across a plurality of nodes;
transmitting the files required to complete each of the jobs to each of the nodes; and
caching certain files at one or more of the nodes across multiple jobs according to a specified cache management policy, the cache management policy dividing divides files into a “shared” group that is to be cached on the nodes across multiple jobs and a “volatile” group that is to be flushed from the nodes after each job.
28. The method as in claim 27 further comprising:
initially tagging a plurality of files as volatile;
moving a first file from the volatile group to a shared group upon detecting that the first file is used in more than one job, wherein volatile files are flushed from node caches after each job and shared files are stored in node caches across multiple jobs.
29. The method as in claim 28 further comprising:
moving the first file back to the volatile group upon detecting that the first file has been modified during the program build.
30. The method as in claim 29 further comprising:
flushing the shared files from each node upon detecting that the first file has been modified.
31. A method comprising:
scheduling jobs for a program build to execute in parallel across a plurality of nodes;
predicting the files required to complete each of the jobs;
preloading the files to each node prior to the execution of each job; and
caching the files at the nodes across multiple jobs according to a specified cache management policy, the cache management policy dividing files into a “shared” group which are to be cached on the nodes across multiple jobs and a “volatile” group which are to be flushed from the nodes after each job.
32. The method as in claim 31 further comprising:
initially tagging a plurality of files as volatile;
moving a first file from the volatile group to a shared group upon detecting that the first file is used in more than one job, wherein volatile files are flushed from node caches after each job and shared files are stored in node caches across multiple jobs.
33. The method as in claim 32 further comprising:
moving the first file back to the volatile group upon detecting that the first file has been modified during the program build.
34. The method as in claim 33 further comprising:
flushing the shared files from each node upon detecting that the first file has been modified.