Scalable event stream data processing using a messaging system
A system processes streaming data and includes at least one processor. The system may write streaming data received from a data source as messages in queues at a queuing cluster. The queuing cluster includes a coordinator node to direct the messages to non-coordinator nodes of the queuing cluster. The system may retrieve the data from the queues based on subscription of topics and store the retrieved data in a consumable repository.
1. A computer-implemented method for processing streaming data, comprising:
writing streaming data received from a data source as messages in queues at a queuing cluster, wherein the queuing duster includes a coordinator node to direct the messages to non-coordinator nodes of the queuing duster;
retrieving the data from the queues based on subscription of topics;
storing the retrieved data in a consumable repository;
listening for the streaming data to be available in the queues by a processing node, the streaming data being event data for one or more events at the data source;
reading each event from the queues by the processing node; and
transforming the events to a data model of the consumable repository.
2. The computer-implemented method of claim 1 , further comprising:
managing the processing node by a coordinator node of a processing cluster; and
retrying processing of the streaming data on a different processing node until successful in response to the processing node encountering a failure.
3. The computer-implemented method of claim 1 , wherein the streaming data includes patient data received from a healthcare network.
4. The computer-implemented method of claim 1 , wherein the consumable repository is a queuing system including a coordinator node to direct the processing node to write the retrieved data to a non-coordinator node in the consumable repository.
5. The computer-implemented method of claim 4 , wherein the queuing cluster and consumable repository each include multiple non-coordinator nodes that provide automated replication and failover through management by the respective coordinator nodes.
6. The computer-implemented method of claim 1 , further comprising one or more from a group of:
routing the retrieved data to different long term storage; and
pushing the retrieved data to a number of different services that satisfy specific business needs.
7. The computer-implemented method of claim 1 , further comprising:
concurrently retrieving data from the consumable repository by a plurality of consumers, wherein the consumers maintain respective pointers to track data that each consumer has processed.