Streaming microservices for stream processing applications
The technology disclosed relates to a system that provides exactly-once processing of stream data. The system includes a queue manager which receives a stream of data. The system establishes aggregation intermediation checkpoints during processing of the received data. To do this, the system partitions delivery of the data stream at offsets, saves partition demarcation offsets at the end of processing windows, and saves intermediate aggregation results to a distributed file system with a window identifier (abbreviated ID) that correlates the offsets and the aggregation results. At each checkpoint, the intermediate aggregation results can be initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme.
1. A computer-implemented method of exactly-once processing stream data, the method including:
receiving a stream of data in a queue manager;
establishing aggregation intermediation checkpoints during processing of the data, including partitioning delivery of the stream data at offsets,
saving partition demarcation offsets at the end of processing windows, and
saving intermediate aggregation results to a distributed file system, including a window identifier (abbreviated ID), correlated offsets for the window, and intermediate results of aggregating over the window, wherein, at each checkpoint, the intermediate aggregation results are initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme;
controlling persistence of key-value data contributing to aggregation on a partition-by-partition basis; and
periodically writing out aggregations to a message queue or to a database, with the writing out governed by a fault tolerance scheme.
2. The computer-implemented method of claim 1 , further including using a processing window ID to link offsets and intermediate aggregation results for a processing window.
3. The computer-implemented method of claim 1 , wherein same or different sets of separate nodes are used to:
read stream data from one or more queues of the queue manager;
establish aggregation intermediation checkpoints during processing of the data; and
periodically write out aggregations to the message queue or to the database.
4. The computer-implemented method of claim 3 , wherein operation of the nodes is connected by processing IDs.
5. The computer-implemented method of claim 1 , wherein a cloud-based service is used for processing stream data.
6. The computer-implemented method of claim 5 , wherein the cloud-based service is at least one of Amazon Web Services™ (abbreviated AWS™), Google Cloud Platform (abbreviated GCP™), Microsoft Azure™, Rackspace™, Joyent™, IBM Cloud Computing™, and OVH™.
7. The computer-implemented method of claim 1 , further including a failure recovery mechanism.
8. The computer-implemented method of claim 1 , wherein the key-value data is saved to a persistence storage database.
9. The computer-implemented method of claim 8 , wherein the persistence storage database is at least one of Apache Cassandra™, PostgreSQL™, ScyllaDB™, ConcourseDB™, RethinkDB™, Oracle Database“, Apache HBase™, Apache Hive™, Google Cloud Datastore™, IBM Information Management System™ (abbreviated IMS), IBM Cloudant™, IBM Compose”, InfluxDB™, DynamoDB™, CockroachDB™, Google BigTable™, Accumulo™, and CouchDB™.
10. The computer-implemented method of claim 1 , further including using one or more write-ahead logs (abbreviated WALs) for fault tolerance.
11. A system including one or more processors coupled to memory, the memory loaded with computer instructions to exactly-once process stream data, the instructions, when executed on the processors, implement actions comprising:
receiving a stream of data in a queue manager;
establishing aggregation intermediation checkpoints during processing of the data, including
partitioning delivery of the stream data at offsets,
saving partition demarcation offsets at the end of processing windows, and
saving intermediate aggregation results to a distributed file system, including a window identifier (abbreviated ID), correlated offsets for the window, and intermediate results of aggregating over the window, wherein, at each checkpoint, the intermediate aggregation results are initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme;
controlling persistence of key-value data contributing to aggregation on a partition-by-partition basis; and
periodically writing out aggregations to a message queue or to a database, with the writing out governed by a fault tolerance scheme.
12. The system of claim 11 , further implementing actions comprising using a processing window ID to link offsets and intermediate aggregation results for a processing window.
13. The system of claim 11 , wherein same or different sets of separate nodes are used to:
read stream data from one or more queues of the queue manager;
establish aggregation intermediation checkpoints during processing of the data; and
periodically write out aggregations to the message queue or to the database.
14. The system of claim 13 , wherein operation of the nodes is connected by processing IDs.
15. The system of claim 11 , further implementing actions comprising a failure recovery mechanism.
16. The system of claim 11 , further implementing actions comprising using one or more write-ahead logs (abbreviated WALs) for fault tolerance.
17. A non-transitory computer readable storage medium impressed with computer program instructions to exactly-once process stream data, the instructions, when executed on a processor, implement a method comprising:
receiving a stream of data in a queue manager;
establishing aggregation intermediation checkpoints during processing of the data, including
partitioning delivery of the stream data at offsets,
saving partition demarcation offsets at the end of processing windows, and
saving intermediate aggregation results to a distributed file system, including a window identifier (abbreviated ID), correlated offsets for the window, and intermediate results of aggregating over the window, wherein, at each checkpoint, the intermediate aggregation results are initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme;
controlling persistence of key-value data contributing to aggregation on a partition-by-partition basis; and
periodically writing out aggregations to a message queue or to a database, with the writing out governed by a fault tolerance scheme.
18. The non-transitory computer readable storage medium of claim 17 , implementing the method further comprising using a processing window ID to link offsets and intermediate aggregation results for a processing window.
19. The non-transitory computer readable storage medium of claim 17 , wherein same or different sets of separate nodes are used to:
read stream data from one or more queues of the queue manager;
establish aggregation intermediation checkpoints during processing of the data; and
periodically write out aggregations to the message queue or to the database.
20. The non-transitory computer readable storage medium of claim 17 , implementing the method further comprising using one or more write-ahead logs (abbreviated WALs) for fault tolerance.