Processes and systems for dataflow routing based on data types and transformations

1. A computer implemented data stream processing service for streaming data input from a source to one or more receivers of a data center in data pipelines, the processing service comprising:

a schema manager that receives commands from a tenant of the data center, the commands contain instructions for parsing the data into one or more data formats and instructions for sending the data in the one or more data formats to the one or more receivers;

a processing services gateway that validates the data and sends the data to a transformation processor;

a transformation processor that receives the data and parses the data into the one or more data formats in accordance with the instructions for parsing the data into one or more data formats; and

an egress service that outputs the data in the one or more data formats to the one or more receivers in accordance with the instructions for sending the data in the one or more data formats to the one or more receivers, each receiver receiving the data in a data pipeline in one of the one or more formats.

2. The processing service of claim 1 further comprises an ingress service that converts the input data into a stream of data, identifies an ingress topic of the data, buffers the input data in a data queue, dequeues the input data, and sends the input data to the processing services gateway.

3. The processing service of claim 1 wherein the schema manager receives the instructions and directs an ingress service to receive the data, directs the processing services gateway to validate the input and send validated data to the transformation processor, and directs the transformation processor to parse the input data into the one or more formats in accordance with the instructions.

4. The processing service of claim 1 wherein the processing services gateway comprises:

an ingress consumer that receives the data from the ingress service and sends the data to one or more transformation engines of the transformation processor in based on ingress topics of the data;

egress producer receives each stream of transformed data output from the transformation engines and send each stream of transformed data separately to the ingress service; and

an egress service that receives the streams of transformed data and sends the transformed data to corresponding egress connectors.

5. The processing service of claim 1 wherein the transformation processor comprises one or more transformation engines, each transformation engine is configured to receive data in a first data format and parses the data into a second data format.

6. The processing service of claim 1 wherein the egress service that receives the data in the one or more formats and outputs the data to one or more egress connectors that forward the data in the one or more formants to the one or more receivers.

7. The processing service of claim 1 wherein the egress service forwards the data output in one of the data formats to an ingress connector that inputs the data to the processing service for parsing into another data format in accordance with the instructions.

8. A process, stored in one or more data-storage devices and executed using one or more processors of a computer system, for routing data in data pipelines from one or more sources to one or more receivers of a data center, the process comprising:

receiving commands for creating dataflows from a tenant of the data center;

sending data input from the one or more sources to one or more transformation engines in accordance with the commands, each transformation engine parsing the data output from one of the one or more sources into a different data format that is different from the data formats produced by the other transformation engines;

sending the data in the different data formats output from each of the transformation engines to one or more egress connectors; and

forwarding the data from the one or more egress connectors to the one or more receivers.

9. The process of claim 8 wherein receiving commands from the tenant of the data center tenant comprises receiving schema, instructions to execute the pipelines in dataflows, and the dataflows in an application programming interface.

10. The process of claim 8 wherein the commands contain instructions for parsing the data into one or more data formats and instructions for sending the data in the one or more data formats to the one or more receivers.

11. The process of claim 8 wherein sending data input from the one or more sources to one or more transformation engines in accordance with the commands comprises:

validating the data and sending the data to one or more transformation engines of a transformation processor based on ingress topics associated with the data;

parses the data into the one or more data formats at the one or more transformation engines in accordance with the instructions; and

outputting the data in the one or more data formats to the one or more receivers in accordance with the instructions for sending the data in the one or more data formats to the one or more receivers, each receiver receiving the data in a data pipeline in one of the one or more formats.

12. The process of claim 8 further comprises:

converting the data into a stream of data;

identifying an ingress topic of the data;

buffering the data in a data queue;

dequeuing the data; and

sending the data to the one or more transformation engines.

13. A computer system for routing data in data pipelines from one or more sources to one or more receivers of a data center, the computer system comprising:

one or more processors;

one or more data-storage devices; and

machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to perform operations comprising:

receiving commands for creating dataflows from a tenant of the data center;

sending data input from the one or more sources to one or more transformation engines in accordance with the commands, each transformation engine parsing the data output from one of the one or more sources into a different data format that is different from the data formats produced by the other transformation engines;

sending the data in the different data formats output from each of the transformation engines to one or more egress connectors; and

forwarding the data from the one or more egress connectors to the one or more receivers.

14. The system of claim 13 wherein receiving commands from the tenant of the data center tenant comprises receiving schema, instructions to execute the pipelines in dataflows, and the dataflows in an application programming interface.

15. The process of claim 13 wherein the commands contain instructions for parsing the data into one or more data formats and instructions for sending the data in the one or more data formats to the one or more receivers.

16. The system of claim 13 wherein sending data input from the one or more sources to one or more transformation engines in accordance with the commands comprises:

validating the data and sending the data to one or more transformation engines of a transformation processor based on ingress topics associated with the data;

parsing the data into the one or more data formats at the one or more transformation engines in accordance with the instructions; and

outputting the data in the one or more data formats to the one or more receivers in accordance with the instructions for sending the data in the one or more data formats to the one or more receivers, each receiver receiving the data in a data pipeline in one of the one or more formats.

17. The system of claim 13 further comprises:

converting the data into a stream of data;

identifying an ingress topic of the data;

buffering the data in a data queue;

dequeuing the data; and

sending the data to the one or more transformation engines.