Allocation of large scale processing job processes to host computing systems
Systems, methods, and software described herein facilitate the allocation of large scale processing jobs to host computing systems. In one example, a method of operating an administration node to allocate processes to a plurality of host computing systems includes identifying a job process for a large scale processing environment (LSPE), and identifying a data repository associated with the job process. The method further includes obtaining data retrieval performance information related to the data repository and the host systems in the LSPE. The method also provides identifying a host system in the host systems for the job process based on the data retrieval performance information, and initiating a virtual node for the job process on the identified host system.
1. An apparatus to allocate job processes to a plurality of host computing systems in a large scale processing environment, the apparatus comprising:
one or more non-transitory computer readable storage media;
processing instructions stored on the one or more non-transitory computer readable storage media that, when executed by a processing system, direct the processing system to at least:
identify a job process for the large scale processing environment;
identify a data repository associated with the job process from a plurality of data repositories, wherein the data repository is stored across a plurality of computing systems;
obtain data retrieval performance information related to the data repository from a cache service executing on each host computing system in the plurality of host computing systems, wherein the data retrieval performance information comprises at least ping time information for accessing the data repository by each cache service, and wherein the cache service acts as an intermediary between virtual nodes executing on a corresponding host computing system and the plurality of data repositories to access data for the virtual nodes;
identify a host computing system in the plurality of host computing systems to execute a virtual node for the job process based on the data retrieval performance information; and
initiate the virtual node on the host computing system for the job process.
2. The apparatus of claim 1 wherein the data retrieval performance information further comprises bandwidth information in accessing the data repository, and physical proximity information to the data repository.
3. The apparatus of claim 1 wherein the job process comprises a Hadoop job process.
4. The apparatus of claim 1 wherein the virtual node for the job process comprises a virtual machine for the job process.
5. The apparatus of claim 1 wherein the virtual node for the job process comprises a virtual container for the job process.
6. The apparatus of claim 1 wherein the processing instructions to identify the data repository associated with the job process direct the processing system to identify a storage location of the data repository associated with the job process.
7. The apparatus of claim 1 wherein the apparatus further comprises the processing system.
8. A method of operating an administration node to allocate job processes to a plurality of host computing systems in a large scale processing environment, the method comprising:
identifying a job process for the large scale processing environment;
identifying a data repository associated with the job process from a plurality of data repositories, wherein the data repository is stored across a plurality of computing systems;
obtaining data retrieval performance information related to the data repository from a cache service executing on each host computing system in the plurality of host computing systems, wherein the data retrieval performance information comprises at least ping time information for accessing the data repository by each cache service, and wherein the cache service acts as an intermediary between virtual nodes executing on a corresponding host computing system and the plurality of data repositories to access data for the virtual nodes;
identifying a host computing system in the plurality of host computing systems to execute a virtual node for the job process based on the data retrieval performance information; and
initiating the virtual node in the host computing system for the job process.
9. The method of claim 8 wherein the data retrieval performance information further comprises bandwidth information in accessing the data repository, and physical proximity information to the data repository.
10. The method of claim 8 wherein the job process comprises an Apache Hadoop job process, an Apache Spark process, or a Disco process.
11. The method of claim 8 wherein the virtual node for the job process comprises a virtual machine for the job process.
12. The method of claim 8 wherein the virtual node for the job process comprises a virtual container for the job process.
13. The method of claim 8 wherein identifying the data repository associated with the job process comprises identifying a storage location of the data repository associated with the job process.
14. The method of claim 8 wherein the plurality of data repositories comprise repositories stored using distributed file systems.
15. A system to allocate job processes amongst a plurality of host computing systems, the system comprising:
an administration node with a first processing system configured to:
identify a job process for the plurality of host computing systems;
identify a data repository from a plurality of data repositories associated with the job process, wherein the data repository is stored across a plurality of computing systems;
transfer a request for data retrieval performance information related to the data repository to each host computing system in the plurality of host computing systems;
the plurality of host computing systems each with a second processing system executing a cache service and configured to:
receive the request;
identify the data retrieval performance information related to the data repository, wherein the data retrieval performance information comprises at least ping time information for accessing the data repository by each cache service, and wherein the cache service acts as an intermediary between virtual nodes executing on a corresponding host computing system and the plurality of data repositories to access data for the virtual nodes;
transfer the data retrieval performance information to the administration node;
the administration node with the processing system further configured to:
receive the data retrieval performance information related to the data repository from each host computing system in the plurality of host computing systems;
identify a host computing system in the plurality of host computing systems to execute a virtual node for the job process based on the data retrieval performance information; and
initiating the virtual node in the host computing system for the job process.
16. The system of claim 15 wherein the data retrieval performance information further comprises bandwidth information in accessing the data repository, and physical proximity information to the data repository.
17. The system of claim 15 wherein the job process comprises an Apache Hadoop job process, an Apache Spark process, or a Disco process.
18. The system of claim 15 wherein the administration node with the first processing system configured to identify a data repository associated with the job process is configured to identify a storage location of the data repository.
19. The system of claim 15 wherein the virtual node for the job process comprises a virtual machine for the job process.