Method and system for a scheduled map executor
Method, system, and programs for data processing. In one example, a record is received. The record is separated into multiple partitions. Each partition is submitted to an iterative pipeline, wherein the iterative pipeline comprises two or more processing stages that are run in parallel and one of the multiple partitions passes through one of the two or more processing stages in one iteration. The multiple partitions are passed through two or more processing stages. The partitions that have passed through each of the two or more processing stages are merged.
1. A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for processing data, the method comprising:
generating a queue of one or more records for processing;
setting a rate at which the one or more records are requested for processing, wherein the rate is determined based on at least one of a network capacity and an availability of processing services;
retrieving the one or more records listed on the queue;
submitting the one or more records to a proxy record reader;
removing the one or more records submitted to the proxy record reader from the queue; and
submitting the one or more records to an actual record reader.
2. The method of claim 1 , further comprising:
polling the queue for a control signal, wherein the control signal is further used to set the rate at which the one or more records are requested for processing.
3. The method of claim 1 , wherein the one or more record are separated into multiple partitions based on a capacity of a network application programming interface.
4. The method of claim 1 , wherein the one or more records are separated into multiple partitions based on a capacity of one or more cloud services.
5. The method of claim 1 , wherein a size of a record of the one or more records determines a number of map tasks to be performed on the one or more records.
6. The method of claim 1 , wherein the rate is further determined based on a size of each record of the one or more records.
7. A system comprising a plurality of nodes, each of which has at least one processor configured for:
generating a queue of one or more records for processing;
setting a rate at which the one or more records are requested for processing, wherein the rate is determined based on at least one of a network capacity and an availability of processing services;
retrieving the one or more records listed on the queue;
submitting the one or more records to a proxy record reader;
removing the one or more records submitted to the proxy record reader from the queue; and
submitting the one or more records to an actual record reader.
8. The system of claim 7 , wherein each node is further configured for:
polling the queue for a control signal, wherein the control signal is further used to set the rate at which the one or more records are requested for processing.
9. The system of claim 7 , wherein the one or more records are separated into multiple partitions based on a capacity of a network application programming interface.
10. The system of claim 7 , wherein a record of the one or more records is separated into multiple partitions based on a capacity of one or more cloud services.
11. The system of claim 7 , wherein a size of the record determines a number of map tasks to be performed on the one or more records.
12. The system of claim 7 , wherein the rate is further determined based on a size of each record of the one or more records.
13. A non-transitory machine-readable medium having information recorded thereon for processing data, wherein the information, when read by the machine, causes the machine to perform the following:
generating a queue of one or more records for processing;
setting a rate at which the one or more records are requested for processing, wherein the rate is determined based on at least one of a network capacity and an availability of processing services;
retrieving the one or more records listed on the queue;
submitting the one or more records to a proxy record reader;
removing the one or more records submitted to the proxy record reader from the queue; and
submitting the one or more records to an actual record reader.
14. The medium of claim 13 , further comprising:
polling the queue for a control signal, wherein the control signal is further used to set the rate at which the one or more records are requested for processing.
15. The medium of claim 13 , wherein the one or more records are separated into multiple partitions based on a capacity of a network application programming interface.
16. The medium of claim 13 , wherein a record of the one or more records is separated into multiple partitions based on a capacity of one or more cloud services.
17. The medium of claim 13 , wherein a size of a record of the one or more records determines a number of map tasks to be performed on the one or more records.
18. The medium of claim 13 , wherein the rate is further determined based on a size of each record of the one or more records.
19. The method of claim 1 , further comprising:
receiving at least a first input file;
separating the first input file into partitions;
determining an input file size of the first input file; and
determining a number of map tasks to be performed based on the input file size, wherein the rate is further determined based on the number.
20. The system of claim 7 , wherein each of the at least one processor is further configured for:
receiving at least a first input file;
separating the first input file into partitions;
determining an input file size of the first input file; and
determining a number of map tasks to be performed based on the input file size, wherein the rate is further determined based on the number.