System and/or method for balancing allocation of data among reduce processes by reallocation
The subject matter disclosed herein relates to a system and/or method for allocating data among reduce processes. In a particular implementation, a map process may be executed to provide intermediate data associating key/value pairs with input data. Intermediate data may be allocated among a plurality of reduce processes. At least a portion of intermediate data initially allocated to one or more of said reduce processes may be re-allocated based, at least in part, on a load factor associated with one or more reduce processes.
1. A method comprising:
executing a map process via a computing platform to provide first intermediate data associating key/value pairs with input data;
allocating said first intermediate data among a plurality of reduce processes, said first intermediate data comprising a first portion and at least a second portion;
re-allocating said at least a second portion of said first intermediate data initially allocated to one or more of said reduce processes based, at least in part, on a load factor associated with said one or more reduce processes, wherein said re-allocating is performed prior to performing said plurality of reduce processes on said at least a portion of said first intermediate data; and
executing said reduce processes on said first portion and said at least a second portion of said first intermediate data, wherein in response to said re-allocating, at least a first helper reduce process is executed on said at least a second portion to generate second intermediate data and at least a second helper reduce process is performed on said second intermediate data.
2. The method of claim 1 , and further comprising determining said load factor based, at least in part, on an estimated time of completion of processing of said portion by said plurality of reduce processes.
3. The method of claim 2 , and further comprising determining said estimated time of completion based, at least in part, on one or more histograms of behavior associated with said at least one of said plurality of reduce processes.
4. The method of claim 3 , wherein at least one of said one or more histograms represents run time per byte.
5. The method of claim 3 , wherein at least one of said one or more histograms represents run time per value of key/value pairs.
6. The method of claim 3 , wherein at least one of said one or more histograms represents run time per key of key/value pairs.
7. The method of claim 3 , wherein at least one of said one or more histograms represents transfer time per byte.
8. The method of claim 3 , wherein at least one of said one or more histograms represents transfer time per value of key/value pairs.
9. The method of claim 3 , wherein at least one of said one or more histograms represents transfer time per key of key/value pairs.
10. The method of claim 1 , wherein said allocating said first intermediate data among said plurality of said reduce processes comprises allocating said first intermediate data based, at least in part, on said key/value pairs.
11. The method of claim 1 , wherein said first or second helper reduce processes execute a function defined by a user as a reduce function.
12. The method of claim 1 , wherein said first or second helper reduce processes execute a function defined by a user as a helper reduce function.
13. The method of claim 1 , wherein said first or second helper reduce processes execute a function defined by a MapReduce system as a helper reduce function.
14. The method of claim 1 , and further comprising: merging output data provided by said second helper reduce process from processing said allocated portion.
15. The method of claim 1 , and further comprising not merging output data provided by said second helper reduce process from processing said allocated portion based upon a user selection.
16. The method of claim 1 , wherein said load factor is based, at least in part, on key-value pairs associated with said portion of first intermediate data.
17. An article comprising:
a non-transitory storage medium having machine-readable instructions stored thereon which are executable by a computing platform to:
initiate execution of a map process to provide first intermediate data associating key/value pairs with input data;
allocate said first intermediate data among a plurality of reduce processes, said first intermediate data comprising a first portion and at least a second portion;
re-allocate said at least a second portion of said first intermediate data initially allocated to one or more of said reduce processes based, at least in part, on a load factor associated with said one or more of said reduce processes, wherein said re-allocation is performed prior to performing said plurality of reduce processes on said at least a portion of said first intermediate data; and
executing said reduce processes on said first portion and said at least a second portion of said first intermediate data, wherein in response to said re-allocating, at least a first helper reduce process is executed on said at least a second portion to generate second intermediate data and at least a second helper reduce process is performed on said second intermediate data.
18. The article of claim 17 , wherein said instructions are further executable by said computing platform to determine said load factor based, at least in part, on an estimated time of completion of processing of said portion by at least one of said plurality of reduce processes.
19. An apparatus comprising:
a computing platform to:
initiate execution of a map process to provide first intermediate data associating key/value pairs with input data;
allocate said first intermediate data among a plurality of reduce processes, said first intermediate data comprising a first portion and at least a second portion;
re-allocate said at least a second portion of said first intermediate data initially allocated to at least one of said reduce processes based, at least in part, on a load factor associated with said at least one reduce process, wherein said re-allocation is performed prior to performing said plurality of reduce processes on said at least a portion of said first intermediate data; and
initiate execution of said reduce processes on said first portion and said at least a second portion of said first intermediate data, wherein in response to said re-allocating, at least a first helper reduce process is executed on said at least a second portion to generate second intermediate data and at least a second helper reduce process is performed on said second intermediate data.
20. The apparatus of claim 19 , said computing platform to determine said load factor based, at least in part, on an estimated time of completion of processing of said portion by at least one of said plurality of reduce processes.
21. The apparatus of claim 20 , said computing platform to determine said estimated time of completion based, at least in part, on one or more histograms of behavior associated with said at least plurality of one or more reduce processes.