Opportunistic job processing using worker processes comprising instances of executable processes created by work order binary code
A global-level manager access a work order from a client and parameters associated with the work order. A service level agreement to meet the work order parameters is determined. The service level agreement includes a price. An indication is received from the client that the service level agreement is accepted. The one or more input files are partitioned into multiple shards, and the work order into multiple jobs. The jobs are distributed among a plurality of clusters to be processed using underutilized computing resources in the clusters. The job outputs are combined to form the work order output. The jobs are monitored to insure that the deadline for completion of the work order will be met.
1. A method for processing data, comprising:
receiving, at a computing system, a work order submitted by a client device, the work order comprising an input file and a binary code;
generating, based on the input file and binary code, one or more jobs;
partitioning the input file into one or more input shards;
processing, using processes running on a group of computers in the computing system, the one or more jobs and input shards, and
wherein processing comprises:
assigning, by a first process, the one or more jobs to a second process, the first process being a higher priority process than the second process,
instantiating, by the second process, one or more native clients, each native client hosting a worker process to perform an assigned work assignment associated with the one or more jobs, each worker process being associated with a given input shard and comprising an instance of at least one executable process created by the binary code,
outputting, by each worker process, one or more output shards to the second process,
providing, by the second process, the one or more output shards to a storage element, and
providing the one or more output shards from the storage element to the client device.
2. The method of claim 1 , wherein the first process comprises a cluster-level manager, the second process comprises a task-level manager and the third process comprises a global-level manager.
3. The method of claim 1 , wherein instantiating comprises creating the one or more native client based on a number of jobs generated.
4. The method of claim 1 , wherein partitioning comprises partitioning the input file into the one or more input shards based on different data types.
5. The method of claim 4 , wherein the different data types are dependent on data size.
6. The method of claim 4 , wherein each different data type is associated with a given one of the prefix values.
7. A system comprising:
one or more processing devices; and
one or more memories storing instructions that, when executed by the one or more processing devices cause the one or more processing devices to:
receive a work order submitted by a client device, the work order comprising an input file and a binary code;
generate, based on the input file and binary code, one or more jobs;
partition the input file into one or more input shards;
process, using processes running on a group of computers that form a computing system, the one or more jobs and one or more input shards;
wherein to process comprises:
assigning, by a first process, the one or more jobs to a second process, the first process being a higher priority process than the second process,
instantiating, by the second process, one or more native clients, each native client hosting a worker process to perform an assigned work assignment associated with the one or more jobs, each worker process being associated with a given input shard and comprising an instance of at least one executable process created by the binary code,
outputting, by each worker process, one or more output shards to the second process,
providing, by the second process, the one or more output shards to a storage element, and
providing the one or more output shards from the storage element to the client device.
8. The system of claim 7 , wherein the first process comprises a cluster-level manager, the second process comprises a task-level manager and the third process comprises a global-level manager.
9. The system of claim 7 , wherein instantiating comprises creating the one or more native clients based on a number of jobs generated.
10. The system of claim 7 , wherein partitioning comprises partitioning the input file into the one or more input shards based on different types of data.
11. The system of claim 10 , wherein the different types of data is dependent on data size.
12. The system of claim 10 , wherein each different type of data is associated with a given one of the prefix values.
13. A non-transitory computer readable medium storing executable instructions, the executable instructions when executed by one or more processing devices cause the one or more processing devices to perform operations comprising:
receiving, at a computing system, a work order submitted by a client device, the work order comprising an input file and a binary code;
generating, based on the input file and binary code, one or more jobs;
partitioning the input file into one or more input shards;
processing, using processes running on a group of computers in the computing system, the one or more jobs and input shards, and
wherein processing comprises:
assigning, by a first process, the one or more jobs to a second process, the first process being a higher priority process than the second process,
instantiating, by the second process, one or more native clients, each native client hosting a worker process to perform an assigned work assignment associated with the one or more jobs, each worker process being associated with a given input shard and comprising an instance of at least one executable process created by the binary code,
outputting, by each worker process, one or more output shards to the second process,
providing, by the second process, the one or more output shards to a storage element, and
providing the one or more output shards from the storage element to the client device.
14. The non-transitory computer readable medium of claim 13 , wherein the first process comprises a cluster-level manager, the second process comprises a task-level manager and the third process comprises a global-level manager.
15. The non-transitory computer readable medium of claim 13 , wherein instantiating comprises creating the one or more native client based on a number of jobs generated.
16. The non-transitory computer readable medium of claim 13 , wherein partitioning comprises partitioning the input file into the one or more input shards based on different types of data.
17. The non-transitory computer readable medium of claim 16 , wherein the different types of data is dependent on data size.
18. The method of claim 1 , wherein the third process distributes the one or more input shards to the first process.
19. The system of claim 7 , wherein the third process distributes the one or more input shards to the first process.
20. The non-transitory computer readable medium of claim 13 , wherein the third process distributes the one or more input shards to the first process.