Compiler-guided software accelerator for iterative HADOOP® jobs
View Patent ↗Various methods are provided directed to a compiler-guided software accelerator for iterative HADOOP® jobs. A method includes identifying intermediate data, generated by an iterative HADOOP® application, below a predetermined threshold size and used less than a predetermined threshold time period. The intermediate data is stored in a memory device. The method further includes minimizing input, output, and synchronization overhead for the intermediate data by selectively using at any given time any one of a Message Passing Interface and Distributed File System as a communication layer. The Message Passing Interface is co-located with the HADOOP® Distributed File System.
1. A method, comprising:
identifying a set of map tasks and reduce tasks capable of being reused across multiple iterations of an iterative HADOOP® application; and
reducing a system load imparted on a computer system executing the iterative HADOOP® application by transforming a source code of the iterative HADOOP® application to launch the map tasks in the set only once and keep the map tasks in the set alive for an entirety of the execution of the iterative HADOOP® application;
wherein the map tasks in the set are kept alive for the entirety of the execution by guarding an invocation to a runjob( ) function beginning at a first iteration of the iterative HADOOP® application to prevent a re-launching of any of the maps tasks and reduce tasks in the set in subsequent iterations of the iterative HADOOP® application, the invocation to the runJob( ) function is guarded by a flag, which is set to true for the first iteration and false for the subsequent iterations.
2. The method of claim 1 , the system load comprises at least one of a cost of re-launching virtual machines, a cost of scheduling tasks in a HADOOP® cluster, and a cost of re-reading invariant input data.
3. The method of claim 1 , wherein the reduce tasks in the set are kept alive for the entirety of the execution by selectively inserting a statement to throw an exception at an end of each reduce task in the set, and wherein a decision to insert the statement is based on a result of a convergence check of a corresponding iteration of the iterative HADOOP® application.
4. The method of claim 1 ,
wherein the map tasks in the set are kept alive for the entirety of the execution by moving code of a map( ) function of the iterative HADOOP® application to a new map( ) function added to at least one map task in the set and storing key/value pairs in a hash table by the new map( ) function, the key/value pairs representing intermediate results produced by map tasks in the set.
5. The method of claim 4 , wherein the map tasks in the set are kept alive for the entirety of the execution by introducing a two-level nested loop in the new map( ) function, the two-level nest loop having an inner loop and an outer loop, wherein the inner loop iterates over the key/value pairs stored in the hash table, and the outer loop supports keeping the map tasks in the set alive for the entirety of the execution.
6. The method of claim 5 , wherein the map tasks in the set are kept alive for the entirety of the execution by inserting a statement in the outer loop to invoke an internal HADOOP® function for flushing and closing a data output stream of the iterative HADOOP® application.
7. The method of claim 5 , wherein the map tasks in the set are kept alive for the entirety of the execution by inserting a statement in the outer loop to wait for a configuration object of a next iteration of the iterative HADOOP® application.
8. A non-transitory storage medium configured with instructions for carrying out the following steps:
identifying a set of map tasks and reduce tasks capable of being reused across multiple iterations of an iterative HADOOP® application; and
reducing a system load imparted on a computer system executing the iterative HADOOP® application by transforming a source code of the iterative HADOOP® application to launch the map tasks in the set only once and keep the map tasks in the set alive for an entirety of the execution of the iterative HADOOP® application;
wherein the map tasks in the set are kept alive for the entirety of the execution by guarding an invocation to a runjob 0 function beginning at a first iteration of the iterative HADOOP® application to prevent a re-launching of any of the maps tasks and reduce tasks in the set in subsequent iterations of the iterative HADOOP® application, the invocation to the runJob( ) function is guarded by a flag, which is set to true for the first iteration and false for the subsequent iterations.