Managing resource allocation in a stream processing framework
The technology disclosed herein relates to method, system, and computer program product (computer-readable storage device) embodiments for managing resource allocation in a stream processing framework. An embodiment operates by configuring an allocation of a task sequence and machine resources to a container, and by running the task sequence, wherein the task sequence is configured to be run continuously as a plurality of units of work corresponding to the task sequence. Some embodiments further include changing the allocation responsive to a determination of an increase in data volume. A query may be taken from the task sequence and processed. Responsive to the query, a real-time result may be returned. Query processing may involve continuously applying a rule to the data stream, in real time or near real time. The rule may be set via a query language. Additionally, the data stream may be partitioned into batches for parallel processing.
1. A method, comprising:
configuring, by at least one computer processor, an allocation of a task sequence and machine resources to a container;
running, by the at least one computer processor, the task sequence, wherein the task sequence is configured to be run continuously as a plurality of units of work multiplexed on the container corresponding to the task sequence;
processing, by the at least one computer processor, a unit of work, of the plurality of units of work, within a window to control data captured from a data stream, wherein the data stream is partitioned into a plurality of batches comprising a time-slice constraint, a batch-size constraint, or a combination thereof; and
automatically changing, by the at least one computer processor, the allocation, based at least in part on the window, responsive to a determination of an increase in data volume of the data stream for processing by the container via the task sequence, wherein the automatically changing the allocation comprises modifying a quantity of the machine resources allocated to the container, wherein the determination of the increase in data volume of the data stream is based on performing a data stream analysis of the data stream and identifying the task sequence to be surging.
2. The method of claim 1 , further comprising processing, by the at least one computer processor, a query for data in the data stream, and wherein the query is taken from the task sequence.
3. The method of claim 2 , further comprising returning, by the at least one computer processor, at least one real-time result, based on the data stream, in response to the query.
4. The method of claim 2 , wherein the processing the query comprises continuously applying a rule to the data stream.
5. The method of claim 4 , wherein the continuously applying the rule to the data stream is performed in real time, wherein the rule is set via a query language.
6. The method of claim 1 , further comprising queuing the plurality of batches, for parallel processing by the container via the machine resources allocated to the container.
7. The method of claim 1 , wherein the task sequence is configured to be run in response to a trigger, the trigger comprising a state transition and a rule, and wherein at least the rule is configured to be set via an interface.
8. A system, comprising:
memory and at least one computer processor coupled to the memory and configured to: configure an allocation of a task sequence and machine resources to a container; run the task sequence, wherein the task sequence is configured to be run continuously as a plurality of units of work multiplexed on the container corresponding to the task sequence;
process a unit of work, of the plurality of units of work, within a window to control data captured from a data stream, wherein the data stream is partitioned into a plurality of batches comprising a time-slice constraint, a batch-size constraint, or a combination thereof; and
automatically change the allocation, based at least in part on the window, responsive to a determination of an increase in data volume of the data stream for processing by the container via the task sequence, wherein the automatically changing the allocation comprises modifying a quantity of the machine resources allocated to the container, wherein the determination of the increase in data volume of the data stream is based on performing a data stream analysis of the data stream and identifying the task sequence to be surging.
9. The system of claim 8 , the at least one computer processor further configured to process a query for data in the data stream, and wherein the query is taken from the task sequence.
10. The system of claim 9 , the at least one computer processor further configured to return at least one real-time result, based on the data stream, in response to the query.
11. The system of claim 9 , wherein to process the query, the at least one computer processor is further configured to apply a rule to the data stream continuously, and wherein to apply the rule to the data stream continuously, the at least one processor is further configured to apply the rule in real time.
12. The system of claim 8 , further comprising queuing the plurality of batches for parallel processing by the container via the machine resources allocated to the container.
13. A non-transitory computer-readable storage device comprising computer-executable instructions that, when executed by at least one computer processor, cause the at least one computer processor to perform operations comprising:
configuring an allocation of a task sequence and machine resources to a container; running the task sequence, wherein the task sequence is configured to be run continuously as a plurality of units of work multiplexed on the container corresponding to the task sequence;
processing a unit of work, of the plurality of units of work, within a window to control data captured from a data stream, wherein the data stream is partitioned into a plurality of batches comprising a time-slice constraint, a batch-size constraint, or a combination thereof; and
automatically changing the allocation, based at least in part on the window, responsive to a determination of an increase in data volume of the data stream for processing by the container via the task sequence, wherein the automatically changing the allocation comprises modifying a quantity of the machine resources allocated to the container, wherein the determination of the increase in data volume of the data stream is based on performing a data stream analysis of the data stream and identifying the task sequence to be surging.
14. The non-transitory computer-readable storage device of claim 13 , the operations further comprising processing a query for data in the data stream, and wherein the query is taken from the task sequence.
15. The non-transitory computer-readable storage device of claim 14 , the operations further comprising returning at least one real-time result, based on the data stream, in response to the query.
16. The non-transitory computer-readable storage device of claim 14 , wherein the processing the query comprises continuously applying a rule to the data stream.
17. The non-transitory computer-readable storage device of claim 16 , wherein the continuously applying the rule to the data stream is performed in real time.
18. The non-transitory computer-readable storage device of claim 16 , wherein the rule is configured to be set via a query language.
19. The non-transitory computer-readable storage device of claim 14 , the operations further comprising queuing the plurality of batches for parallel processing by the container via the machine resources allocated to the container.