Transfer of data streaming services to provide continuous data flow
Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.
1. An apparatus comprising:
a processor to process data for streaming to one or more organizations; and
a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide:
a centralized work distribution service to track status of each of a plurality of data streams to the one or more organization,
a plurality of nodes, each node being a virtual machine having capability to service a plurality of data streams to the one or more organizations, and
a database to store a state for each data stream serviced by any node of the plurality of nodes; and
wherein each node of the plurality of nodes including:
a first daemon service to monitor connectivity of the node to a plurality of dependency services for the node and, upon detecting a loss of connection to any of the plurality of dependency services for the node, the node to discontinue ownership of all data streams being serviced by the node; and
a second daemon service to poll the centralized work distribution service for data streams that are not assigned, wherein, upon the node obtaining ownership of one or more unassigned data streams, the node is to access the database to determine a starting point for each of the one or more data streams.
2. The apparatus of claim 1 , wherein the servicing of a data stream to an organization is limited to a single node of the plurality of nodes at any time.
3. The apparatus of claim 1 , wherein the nodes of the plurality of nodes do not communicate with each other.
4. The apparatus of claim 1 , wherein each node of the plurality of nodes operates independently from the other nodes of the plurality of nodes.
5. The apparatus of claim 1 , wherein the plurality of nodes exists in a cloud computing platform.
6. The apparatus of claim 1 , wherein the plurality of dependency services include one or more of a database service, a coordination service, and a cache service.
7. The apparatus of claim 1 , wherein one or more data streams of each node including data representing changes occurring on a customer relationship management (CRM) system.
8. A non-transitory computer-readable storage medium having stored thereon data representing sequences of instructions that, when executed by a processor, cause the processor to perform operations comprising:
tracking of a status of one or more data streaming jobs for one or more organizations by a central work distribution service, the one or more data streaming jobs being serviced by a plurality of nodes, each node being a virtual machine having capability to service a plurality of data streams to the one or more organizations;
storing in a database a state for each of the one or more data streaming jobs serviced by the plurality of nodes;
polling of the central work distribution service for available data streaming jobs by a first daemon service of each of the plurality of nodes, the central work distribution service to assign available data streaming jobs to the nodes in response to the polling;
upon a node being assigned one or more data streaming jobs by the central work distribution service, accessing the database by the node to determine a starting point for each of the one or more data streams;
monitoring of a plurality of dependency services of each of the plurality of nodes by a second daemon service of each node; and
upon the second daemon service of a first node detecting loss of connectivity with any of the plurality of dependency services for the first node, discontinuing ownership by the first node of all data streaming jobs being serviced by the first node, the central work distribution service changing the status of all data streaming jobs of the first node to available.
9. The medium of claim 8 , wherein the servicing of a data streaming job to an organization is limited to a single node of the plurality of nodes at any time.
10. The medium of claim 8 , wherein the nodes of the plurality of nodes do not communicate with each other.
11. The medium of claim 8 , further comprising instructions that, when executed by the processor, cause the processor to perform operations comprising:
each node of the plurality of nodes operating independently from the other nodes of the plurality of nodes.
12. The medium of claim 8 , wherein the plurality of nodes exists in a cloud computing platform.
13. The medium of claim 8 , wherein the plurality of dependency services include one or more of database, coordination, and cache services.
14. The medium of claim 8 , wherein one or more data streaming jobs of each node include one or more streams of data representing changes in a customer relationship management (CRM) system.
15. A system comprising:
data storage for system data and tenant data;
a processor system to process data for streaming to one or more organizations;
a network interface to provide connection with one or more user systems; and
a memory to store data for streaming to the one or more organizations, wherein the system is to provide:
a centralized work distribution service to track status of each of a plurality of data streams to the one or more organization,
a plurality of nodes, each node being a virtual machine having capability to service a plurality of data streams to the one or more organizations, and
a database to store a state for each data stream serviced by any node of the plurality of nodes;
wherein each node of the plurality of nodes including:
a first daemon service to monitor connectivity of the node to a plurality of dependency services for the node and, upon detecting a loss of connection to any of the plurality of dependency services for the node, the node to discontinue ownership of all data streams being serviced by the node; and
a second daemon service to poll the centralized work distribution service for data streams that are not assigned, wherein, upon the node obtaining ownership of one or more unassigned data streams, the node is to access the database to determine a starting point for each of the one or more data streams.
16. The system of claim 15 , wherein the servicing of a data stream to an organization is limited to a single node of the plurality of nodes at any time.
17. The system of claim 15 , wherein the nodes of the plurality of nodes operate independently from each other and do not share information with each other.
18. The system of claim 15 , wherein the plurality of nodes exists in a cloud computing platform.
19. The system of claim 15 , wherein the plurality of dependency services include one or more of database, coordination, and cache services.
20. The apparatus of claim 1 , wherein operation of the plurality of dependency services is required for operation of the node.