Stream-based job processing
Systems and techniques for managing and executing digital workflows are described. A technique described includes obtaining a job record from a job queue from a first server; assigning a node associated with a second server to handle a task indicated by the job record; operating, at the second server, a first action block in the node to produce output results in response to executing the task and to forward the output results to batch blocks; operating, at the second server, the batch blocks in the node to respectively accumulate different batch groups of the output results; operating, at the second server, the batch blocks in the node to respectively forward the different batch groups of the output results to respective second action blocks; and operating, at the second server, the second action blocks in the node to respectively process the different batch groups of the output results.
1. A computer-implemented method comprising:
obtaining, from a first server system, a job record from a job queue;
assigning a node associated with a second server system to handle a task indicated by the job record;
operating, at the second server system, a first action block in the node to produce a plurality of output results in response to executing the task and to forward the plurality of output results to batch blocks;
operating, at the second server system, the batch blocks in the node to respectively accumulate batch groups of the output results, the batch groups being formed based on a mapping template that defines protocols for aggregating sets of the output results, the sets of the output results corresponding to different respective layers;
operating, at the second server system, the batch blocks in the node to forward, to second action blocks, the batch groups of the output results meeting a predetermined data amount threshold, each of the batch blocks transmitting the output results to a respective second action block of the second action blocks that responded to a polling action by indicating an availability to process the batch groups of the output results; and
operating, at the second server system, the second action blocks in the node to respectively process the batch groups of the output results.
2. The computer-implemented method of claim 1 , wherein the batch groups of the output results are first batch groups of the output results, wherein the method comprises:
operating, at the second server system, the batch blocks in the node to respectively accumulate different second batch groups of the output results;
operating, at the second server system, the batch blocks in the node to respectively forward the different second batch groups of the output results to the second action blocks; and
operating, at the second server system, the second action blocks in the node to respectively process the different second batch groups of the output results.
3. The computer-implemented method of claim 2 , comprising:
operating, at the second server system, the batch blocks to generate a signal to throttle forwarding of output results from the plurality of output results from the first action block.
4. The computer-implemented method of claim 3 , wherein the signal is generated based on forwarding a predetermined number of output results to a second action block.
5. The computer-implemented method of claim 1 , wherein operating the second action blocks in the node to respectively process the batch groups of the output results comprises operating the second action blocks to respectively upload the batch groups of the output results to a data store associated with the first server system.
6. The computer-implemented method of claim 1 , wherein the job record comprises an attachment identifier, wherein the method comprises:
operating, at the second server system, the first action block in the node to retrieve one or more files associated with the attachment identifier.
7. The computer-implemented method of claim 1 , comprising:
operating, at the second server system, a link block in the node to copy the plurality of output results from the first action block to the batch blocks.
8. A non-transitory computer-readable storage medium comprising a program for execution by a processor of a device, the program including instructions which, when executed by the processor, cause the device to perform operations comprising:
obtaining, from a first server system, a job record from a job queue;
assigning a node associated with a second server system to handle a task indicated by the job record;
operating, at the second server system, a first action block in the node to produce a plurality of output results in response to executing the task and to forward the plurality of output results to batch blocks;
operating, at the second server system, the batch blocks in the node to respectively accumulate batch groups of the output results, the batch groups being formed based on a mapping template that defines protocols for aggregating sets of the output results, the sets of the output results corresponding to different respective layer;
operating, at the second server system, the batch blocks in the node to forward, to second action blocks, the batch groups of the output results meeting a predetermined data amount threshold, each of the batch blocks transmitting the output results to a respective second action block of the second action blocks that responded to a polling action by indicating an availability to process the batch groups of the output results; and
operating, at the second server system, the second action blocks in the node to respectively process the batch groups of the output results.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the batch groups of the output results are first batch groups of the output results, wherein the operations comprise:
operating, at the second server system, the batch blocks in the node to respectively accumulate different second batch groups of the output results;
operating, at the second server system, the batch blocks in the node to respectively forward the different second batch groups of the output results to the second action blocks; and
operating, at the second server system, the second action blocks in the node to respectively process the different second batch groups of the output results.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the operations comprise:
operating, at the second server system, the batch blocks to generate a signal to throttle forwarding of output results from the plurality of output results from the first action block.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the signal is generated based on forwarding a predetermined number of output results to a second action block.
12. The non-transitory computer-readable storage medium of claim 8 , wherein operating the second action blocks in the node to respectively process the batch groups of the output results comprises operating the second action blocks to respectively upload the batch groups of the output results to a data store associated with the first server system.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the job record comprises an attachment identifier, wherein the operations comprise:
operating, at the second server system, the first action block in the node to retrieve one or more files associated with the attachment identifier.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the operations comprise:
operating, at the second server system, a link block in the node to copy the plurality of output results from the first action block to the batch blocks.
15. A system comprising:
a processor; and
a computer-readable storage medium coupled with the processor, wherein the computer-readable storage medium comprising a program for execution by the processor, the program including instructions which, when executed by the processor, cause the processor to perform operations comprising:
obtaining a job record from a job queue;
assigning a node to handle a task indicated by the job record;
operating a first action block in the node to produce a plurality of output results in response to executing the task and to forward the plurality of output results to batch blocks;
operating the batch blocks in the node to respectively accumulate batch groups of the output results, the batch groups being formed based on a mapping template that defines protocols for aggregating sets of the output results, the sets of the output results corresponding to different respective layer;
operating the batch blocks in the node to forward, to second action blocks, the batch groups of the output results meeting a predetermined data amount threshold, each of the batch blocks transmitting the output results to a respective second action block of the second action blocks that responded to a polling action by indicating an availability to process the batch groups of the output results; and
operating the second action blocks in the node to respectively process the batch groups of the output results.
16. The system of claim 15 , wherein the batch groups of the output results are first batch groups of the output results, wherein the operations comprise:
operating the batch blocks in the node to respectively accumulate different second batch groups of the output results;
operating the batch blocks in the node to respectively forward the different second batch groups of the output results to the second action blocks; and
operating the second action blocks in the node to respectively process the different second batch groups of the output results.
17. The system of claim 16 , wherein the operations comprise:
operating the batch blocks to generate a signal to throttle forwarding of output results from the plurality of output results from the first action block.
18. The system of claim 17 , wherein the signal is generated based on forwarding a predetermined number of output results to a second action block.
19. The system of claim 15 , wherein operating the second action blocks in the node to respectively process the batch groups of the output results comprises operating the second action blocks to respectively upload the batch groups of the output results to a data storage system.
20. The system of claim 15 , wherein the job record comprises an attachment identifier, wherein the operations comprise:
operating the first action block in the node to retrieve one or more files associated with the attachment identifier.
21. The system of claim 15 , wherein the operations comprise:
operating a link block in the node to copy the plurality of output results from the first action block to the batch blocks.