Recovery strategy for a stream processing system
The technology disclosed relates to discovering multiple previously unknown and undetected technical problems in fault tolerance and data recovery mechanisms of modem stream processing systems. In addition, it relates to providing technical solutions to these previously unknown and undetected problems. In particular, the technology disclosed relates to discovering the problem of modification of batch size of a given batch during its replay after a processing failure. This problem results in over-count when the input during replay is not a superset of the input fed at the original play. Further, the technology disclosed discovers the problem of inaccurate counter updates in replay schemes of modem stream processing systems when one or more keys disappear between a batch's first play and its replay. This problem is exacerbated when data in batches is merged or mapped with data from an external data store.
1. A method of data processing comprising:
assigning, by an internet of things (IoT) platform, priority levels to one or more input pipelines configured for receiving data from one or more input data streams;
receiving, at the IoT platform, and via the one or more input pipelines, data from the one or more input data streams and transforming the received data for processing;
processing the transformed data in two or more batches according to the priority levels assigned to the one or more input pipelines using a plurality of workers in a container of the IoT platform, wherein processing for a first batch is performed such that a first worker of the plurality of workers completes a first stage of processing of the first batch before a second worker of the plurality of workers begins a second stage of processing of the first batch according to a dependency rule defined in the container; and
monitoring performance metrics of the IoT platform and one or more applications supported by the IoT platform, the performance metrics being determined based at least on one or more application records for the one or more applications supported by the IoT platform.
2. The method of claim 1 , further comprising:
allocating the one or more input pipelines to the container of the IoT platform.
3. The method of claim 2 , further comprising:
multiplexing the one or more input pipelines over the plurality of workers in the container.
4. The method of claim 2 , further comprising:
providing a shared connection between the one or more input pipelines and at least one common resource external to the container, wherein the one or more input pipelines have connections to the at least one common resource external to the container.
5. The method of claim 1 , further comprising:
assigning a unique event offset to each event of the two or more batches;
identifying the first batch of the two or more batches by determining a current event offset and an end event offset such that the first batch includes event logged between the current event offset and the end event offset; and
processing the first batch using a computing grid that consumes data from a message bus that queues events from the one or more input data streams in a plurality of batches including the two or more batches.
6. The method of claim 5 , further comprising:
detecting failed processing of the first batch;
determining that the end event offset is subject to change caused by the message bus queuing new events before or during reprocessing of the first batch, wherein the reprocessing is performed in response to detecting the failed processing of the first batch; and
restricting the reprocessing of the first batch to events logged between the current event offset and the end event offset, thereby preventing inclusion of the new events in the first batch.
7. The method of claim 1 , wherein the one or more application records comprises a plurality of contact records, a plurality of account records, other records, or a combination thereof.
8. An apparatus for data processing comprising:
a processor,
memory in electronic communication with the processor, and
instructions stored in the memory and executable by the processor to cause the apparatus to:
assign, by an internet of things (IoT) platform, priority levels to one or more input pipelines configured for receiving data from one or more input data streams;
receive, at the IoT platform, and via the one or more input pipelines, data from the one or more input data streams and transforming the received data for processing;
process the transformed data in two or more batches according to the priority levels assigned to the one or more input pipelines using a plurality of workers in a container of the IoT platform, wherein processing for a first batch is performed such that a first worker of the plurality of workers completes a first stage of processing of the first batch before a second worker of the plurality of workers begins a second stage of processing of the first batch according to a dependency rule defined in the container; and
monitor performance metrics of the IoT platform and one or more applications supported by the IoT platform, the performance metrics being determined based at least on one or more application records for the one or more applications supported by the IoT platform.
9. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
allocate the one or more input pipelines to the container of the IoT platform.
10. The apparatus of claim 9 , wherein the instructions are further executable by the processor to cause the apparatus to:
multiplex the one or more input pipelines over the plurality of workers in the container.
11. The apparatus of claim 9 , wherein the instructions are further executable by the processor to cause the apparatus to:
provide a shared connection between the one or more input pipelines and at least one common resource external to the container, wherein the one or more input pipelines have connections to the at least one common resource external to the container.
12. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
assign a unique event offset to each event of the two or more batches;
identify the first batch of the two or more batches by determining a current event offset and an end event offset such that the first batch includes event logged between the current event offset and the end event offset; and
process the first batch using a computing grid that consumes data from a message bus that queues events from the one or more input data streams in a plurality of batches including the two or more batches.
13. The apparatus of claim 12 , wherein the instructions are further executable by the processor to cause the apparatus to:
detect failed processing of the first batch;
determine that the end event offset is subject to change caused by the message bus queuing new events before or during reprocessing of the first batch, wherein the reprocessing is performed in response to detecting the failed processing of the first batch; and
restrict the reprocessing of the first batch to events logged between the current event offset and the end event offset, thereby preventing inclusion of the new events in the first batch.
14. A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by a processor to:
assign, by an internet of things (IoT) platform, priority levels to one or more input pipelines configured for receiving data from one or more input data streams;
receive, at the IoT platform, and via the one or more input pipelines, data from the one or more input data streams and transforming the received data for processing;
process the transformed data in two or more batches according to the priority levels assigned to the one or more input pipelines using a plurality of workers in a container of the IoT platform, wherein processing for a first batch is performed such that a first worker of the plurality of workers completes a first stage of processing of the first batch before a second worker of the plurality of workers begins a second stage of processing of the first batch according to a dependency rule defined in the container; and
monitor performance metrics of the IoT platform and one or more applications supported by the IoT platform, the performance metrics being determined based at least on one or more application records for the one or more applications supported by the IoT platform.
15. The non-transitory computer-readable medium of claim 14 , wherein the instructions are further executable by the processor to:
allocate the one or more input pipelines to the container of the IoT platform.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
multiplex the one or more input pipelines over the plurality of workers in the container.
17. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
provide a shared connection between the one or more input pipelines and at least one common resource external to the container, wherein the one or more input pipelines have connections to the at least one common resource external to the container.
18. The non-transitory computer-readable medium of claim 14 , wherein the instructions are further executable by the processor to:
assign a unique event offset to each event of the two or more batches;
identify the first batch-of the two or more batches by determining a current event offset and an end event offset such that the first batch includes event logged between the current event offset and the end event offset; and
process the first batch using a computing grid that consumes data from a message bus that queues events from the one or more input data streams in a plurality of batches including the two or more batches.
19. The non-transitory computer-readable medium of claim 18 , wherein the instructions are further executable by the processor to:
detect failed processing of the first batch;
determine that the end event offset is subject to change caused by the message bus queuing new events before or during reprocessing of the first batch, wherein the reprocessing is performed in response to detecting the failed processing of the first batch; and
restrict the reprocessing of the first batch to events logged between the current event offset and the end event offset, thereby preventing inclusion of the new events in the first batch.