Systems and methods for distributing a workplan for data flow execution based on an arbitrary graph describing the desired data flow
Various embodiments of the present invention are directed to the creation of multiple redundant chains of transforms, each on a separate processing thread, for a data flow execution (DFE) of a data transformation pipeline (DTP). For certain of these embodiments, a “distributor” receives a buffer as input and directs that buffer to one of several parallel identical threads to process that buffer. A scheduler would create each of these multiple threads, each thread having an identical (redundant) strings of transforms (chains) downstream from the distributor, and all of which would lead even further downstream to a collector that is responsible for collecting and, if necessary, ordering the buffers processed by the previous redundant chains. In this way, the distributors and collectors provide increased scalability for the pipeline by implicitly partitioning (distributing) individual buffers to one of many threads for at least a part of their execution/processing.
1. An automated processor-implemented method for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said automated method comprising:
automatically translating and optimizing, via the processor, a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one replica of said at least one chain of distributable transforms;
automatically adding to the chain of transforms a distributor immediately upstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms; and
automatically adding to the chain of transforms a collector immediately downstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms.
2. The method of claim 1 further comprising:
said distributor receiving a set of buffers;
said distributor distributing a first subset of said set of buffers to said at least one chain of distributable transforms; and
said distributor distributes a second subset of said set of buffers to said at least one replica of said at least one chain of distributable transforms.
3. The method of claim 1 further comprising:
said collector receiving a first subset of buffers from said at least one chain of distributable transforms;
said collector receiving a second subset of buffers from said at least one replica of said at least one chain of distributable transforms; and
said collector sending a set of buffers to a component downstream from said collector, said set of buffers comprising said first subset of buffers and said second subset of buffers.
4. The method of claim 3 , wherein said set of buffers comprise an order, said method further comprising said collector reordering said set of buffers based on said order.
5. The method of claim 1 wherein said at least one chain of distributable transforms executes as a first thread and said at least one replica of said at least one chain of distributable transforms executes as a second thread.
6. An automated processor-implemented method for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said automated method comprising:
automatically translating and optimizing, via the processor, a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one redundant replica of said at least one chain of distributable transforms;
for a set of buffers comprising a plurality of individual buffers, automatically distributing each individual buffer to either said at least one chain of distributable transforms or said at least one redundant replica of said at least one chain of distributable transforms.
7. An automated system comprising a processor for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said automated system comprising at least one subsystem for:
automatically translating and optimizing, via the processor, a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one replica of said at least one chain of distributable transforms;
automatically adding to the chain of transforms a distributor immediately upstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms; and
automatically adding to the chain of transforms a collector immediately downstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms.
8. The system of claim 7 further comprising at least one subsystem for:
said distributor receiving a set of buffers;
said distributor distributing a first subset of said set of buffers to said at least one chain of distributable transforms; and
said distributor distributes a second subset of said set of buffers to said at least one replica of said at least one chain of distributable transforms.
9. The system of claim 7 further comprising at least one subsystem for:
said collector receiving a first subset of buffers from said at least one chain of distributable transforms;
said collector receiving a second subset of buffers from said at least one replica of said at least one chain of distributable transforms; and
said collector sending a set of buffers to a component downstream from said collector, said set of buffers comprising said first subset of buffers and said second subset of buffers.
10. The system of claim 9 , wherein said set of buffers comprise an order, said method further comprising at least one subsystem for said collector to reorder said set of buffers based on said order.
11. The system of claim 7 further comprising at least one subsystem whereby said at least one chain of distributable transforms executes as a first thread and said at least one replica of said at least one chain of distributable transforms executes as a second thread.
12. An automated system for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said automated system comprising a processor and at least one subsystem for:
automatically translating and optimizing a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one redundant replica of said at least one chain of distributable transforms;
for a set of buffers comprising a plurality of individual buffers, automatically distributing each individual buffer to either said at least one chain of distributable transforms or said at least one redundant replica of said at least one chain of distributable transforms.
13. A non-transitory computer-readable storage medium comprising computer-readable instructions for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said computer-readable instructions comprising instructions for:
automatically translating and optimizing a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one replica of said at least one chain of distributable transforms;
automatically adding to the chain of transforms a distributor immediately upstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms; and
automatically adding to the chain of transforms a collector immediately downstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms.
14. The non-transitory computer-readable storage medium of claim 13 further comprising instructions for:
said distributor receiving a set of buffers;
said distributor distributing a first subset of said set of buffers to said at least one chain of distributable transforms; and
said distributor distributes a second subset of said set of buffers to said at least one replica of said at least one chain of distributable transforms.
15. The non-transitory computer-readable storage medium of claim 13 further comprising instructions for:
said collector receiving a first subset of buffers from said at least one chain of distributable transforms;
said collector receiving a second subset of buffers from said at least one replica of said at least one chain of distributable transforms; and
said collector sending a set of buffers to a component downstream from said collector, said set of buffers comprising said first subset of buffers and said second subset of buffers.
16. The non-transitory computer-readable storage medium of claim 15 , wherein said set of buffers comprise an order, said method further comprising instructions whereby said collector reorders said set of buffers based on said order.
17. The non-transitory computer-readable storage medium of claim 13 further comprising instructions whereby said at least one chain of distributable transforms executes as a first thread and said at least one replica of said at least one chain of distributable transforms executes as a second thread.
18. A non-transitory computer-readable storage medium comprising computer-readable instructions for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said computer-readable instructions comprising instructions for:
automatically translating and optimizing a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one redundant replica of said at least one chain of distributable transforms;
for a set of buffers comprising a plurality of individual buffers, automatically distributing each individual buffer to either said at least one chain of distributable transforms or said at least one redundant replica of said at least one chain of distributable transforms.
19. A processor for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said processor:
automatically translating and optimizing a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one replica of said at least one chain of distributable transforms;
automatically adding to the chain of transforms a distributor immediately upstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms; and
automatically adding to the chain of transforms a collector immediately downstream from, and coupled to, said at least one chain of distributable transforms and said at least one replica of said at least one chain of distributable transforms.
20. The processor of claim 19 wherein:
said distributor receives a set of buffers;
said distributor distributes a first subset of said set of buffers to said at least one chain of distributable transforms; and
said distributor distributes a second subset of said set of buffers to said at least one replica of said at least one chain of distributable transforms.
21. The processor of claim 19 wherein:
said collector receives a first subset of buffers from said at least one chain of distributable transforms;
said collector receives a second subset of buffers from said at least one replica of said at least one chain of distributable transforms; and
said collector sends a set of buffers to a component downstream from said collector, said set of buffers comprising said first subset of buffers and said second subset of buffers.
22. The processor of claim 21 , wherein said set of buffers comprise an order, and said collector reorders said set of buffers based on said order.
23. The processor of claim 19 wherein said at least one chain of distributable transforms executes as a first thread and said at least one replica of said at least one chain of distributable transforms executes as a second thread.
24. A processor for automatically distributing a workplan for a data flow execution (DFE), said DFE comprising a chain of transforms having at least one chain of distributable transforms, said processor:
automatically translating and optimizing a graph describing a desired data flow into the chain of transforms;
automatically distinguishing non-distributable and distributable transforms in the chain of transforms including the at least one chain of distributable transforms;
automatically scaling the chain of transforms for execution by:
automatically adding to the chain of transforms at least one redundant replica of said at least one chain of distributable transforms;
for a set of buffers comprising a plurality of individual buffers, automatically distributing each individual buffer to either said at least one chain of distributable transforms or said at least one redundant replica of said at least one chain of distributable transforms.