Data processing and data movement in cloud computing environment

1. A method for moving data from a source site to a target site in a cloud computing platform, comprising:

receiving a plurality of data sets to be moved from the source site to the target site at a plurality of containerized data ingest components located at the source site;

providing the received plurality of data sets from the plurality of data ingest components to a staging cluster comprising a plurality of containerized broker components located at the source site, wherein the plurality of containerized broker components queue the plurality of data sets, wherein the staging cluster replicates one or more partitions of the received data set between broker components, and wherein each broker component performs a data deduplication operation;

providing the queued plurality of data sets from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components, wherein the plurality of containerized data processing components process the plurality of data sets, wherein the processing stage performs one or more of data encryption, data reduction, and data indexing prior to a data set being transmitted to the target site;

transmitting the plurality of data sets from the plurality of containerized data processing components to the target site;

wherein, for each data ingest component of the plurality of data ingest components, a respective pipeline is formed through the staging cluster and the processing cluster, and wherein the staging cluster and the processing cluster are scalable such that the method further comprises:

adding an additional pipeline comprising a given containerized broker component in the staging cluster and a given containerized data processing component in the processing cluster when a data ingest component is added;

removing an existing pipeline comprising a given containerized broker component in the staging cluster and a given containerized data processing component in the processing cluster when an existing data ingest component is removed;

receiving the transmitted plurality of data sets at a staging cluster comprising a plurality of containerized broker components located at the target site, wherein the plurality of containerized broker components queue the plurality of data sets;

providing the queued plurality of data sets from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components; and

providing the plurality of processed data sets from the plurality of containerized data processing components to a plurality of data application components;

wherein, for each data set received, a respective pipeline is formed through the staging cluster and the processing cluster, and wherein the staging cluster and the processing cluster are scalable to add or remove a pipeline depending on the number of data sets received;

wherein the source site and the target site are implemented via one or more processing devices operatively coupled via a communication network.

2. The method of claim 1 , wherein each data ingest component receives a data moving task associated with a received data set.

3. The method of claim 2 , wherein each data ingest component processes the received data set in accordance with one or more parameters of the data moving task.

4. The method of claim 3 , wherein the one or more parameters of the data moving task specify one or more of: a source type of the received data set; an identifier of the source of the received data set, target site information; and one or more processes to be performed on the received data set.

5. The method of claim 3 , wherein each data ingest component launches one or more worker threads to read the received data set.

6. The method of claim 5 , wherein each data ingest component partitions the received data set and sends the partitioned data set to the staging cluster.

7. The method of claim 6 , wherein each data ingest component partitions the received data set in accordance with one of a content key and an offset key.

8. The method of claim 1 , wherein each data ingest component is configured to execute in accordance with an instance of an application program.

9. The method of claim 8 , wherein the instance of the application program is one of a rewritable application instance and a read-only application instance.

10. The method of claim 1 , wherein each broker component performs an asynchronous data flush operation.

11. The method of claim 1 , wherein the staging cluster and the processing cluster form a directed acyclic graph structure.

12. The method of claim 1 , wherein the staging cluster and the processing cluster perform a two-phase acknowledgment procedure.

13. The method of claim 12 , wherein the two-phase acknowledgment procedure comprises an acknowledge step and a commit step to confirm that a data set has been fully processed by the processing cluster.

14. The method of claim 13 , wherein the staging cluster removes the data set once receiving confirmation that the data set has been fully processed.

15. The method of claim 1 , wherein each of the data application components is executed in its own container.

16. A system for moving data from a source site to a target site in a cloud computing platform, the system comprising:

at least one processor, coupled to a memory, and configured to:

receive a plurality of data sets to be moved from the source site to the target site at a plurality of containerized data ingest components located at the source site;

provide the received plurality of data sets from the plurality of data ingest components to a staging cluster comprising a plurality of containerized broker components located at the source site, wherein the plurality of containerized broker components queue the plurality of data sets, wherein the staging cluster replicates one or more partitions of the received data set between broker components, and wherein each broker component performs a data deduplication operation;

provide the queued plurality of data sets from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components, wherein the plurality of containerized data processing components process the plurality of data sets, wherein the processing stage performs one or more of data encryption, data reduction, and data indexing prior to a data set is transmitted to the target site;

transmit the plurality of data sets from the plurality of containerized data processing components to the target site;

wherein, for each data ingest component of the plurality of data ingest components, a respective pipeline is formed through the staging cluster and the processing cluster, and wherein the staging cluster and the processing cluster are scalable such that the processor is further configured to:

add an additional pipeline comprising a given containerized broker component in the staging cluster and a given containerized data processing component in the processing cluster when a data ingest component is added;

remove an existing pipeline comprising a given containerized broker component in the staging cluster and a given containerized data processing component in the processing cluster when an existing data ingest component is removed;

receive the transmitted plurality of data sets at a staging cluster comprising a plurality of containerized broker components located at the target site, wherein the plurality of containerized broker components queue the plurality of data sets;

provide the queued plurality of data sets from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components; and

provide the plurality of processed data sets from the plurality of containerized data processing components to a plurality of data application components;

wherein, for each data set received, a respective pipeline is formed through the staging cluster and the processing cluster, and wherein the staging cluster and the processing cluster are scalable to add or remove a pipeline depending on the number of data sets received;

wherein the source site and the target site are operatively coupled via a communication network.

17. The system of claim 16 , wherein each data ingest component receives a data moving task associated with a received data set.

18. The system of claim 16 , wherein each of the data application components is executed in its own container.

19. An article of manufacture for moving data from a source site to a target site in a cloud computing platform, the article of manufacture comprising a non-transitory processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device implement the steps of:

receiving a plurality of data sets to be moved from the source site to the target site at a plurality of containerized data ingest components located at the source site;

providing the received plurality of data sets from the plurality of data ingest components to a staging cluster comprising a plurality of containerized broker components located at the source site, wherein the plurality of containerized broker components queue the plurality of data sets, wherein the staging cluster replicates one or more partitions of the received data set between broker components, and wherein each broker component performs a data deduplication operation;

providing the queued plurality of data sets from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components, wherein the plurality of containerized data processing components process the plurality of data sets, wherein the processing stage performs one or more of data encryption, data reduction, and data indexing prior to a data set is transmitted to the target site;

transmitting the plurality of data sets from the plurality of containerized data processing components to the target site;

wherein, for each data ingest component of the plurality of data ingest components, a respective pipeline is formed through the staging cluster and the processing cluster, and wherein the staging cluster and the processing cluster are scalable such that the implemented steps further comprise:

adding an additional pipeline comprising a given containerized broker component in the staging cluster and a given containerized data processing component in the processing cluster when a data ingest component is added;

removing an existing pipeline comprising a given containerized broker component in the staging cluster and a given containerized data processing component in the processing cluster when an existing data ingest component is removed;

receiving the transmitted plurality of data sets at a staging cluster comprising a plurality of containerized broker components located at the target site, wherein the plurality of containerized broker components queue the plurality of data sets;

providing the queued plurality of data sets from the plurality of containerized broker components to a processing cluster comprising a plurality of containerized data processing components; and

providing the plurality of processed data sets from the plurality of containerized data processing components to a plurality of data application components;

wherein, for each data set received, a respective pipeline is formed through the staging cluster and the processing cluster, and wherein the staging cluster and the processing cluster are scalable to add or remove a pipeline depending on the number of data sets received;

wherein the source site and the target site operatively coupled via a communication network.

20. The article of manufacture of claim 19 , wherein each of the data application components is executed in its own container.