Seamless data movement and metadata management in a hybrid cloud setting using a configurable micro services based architecture

1. A method, comprising:

receiving, by a computer-based system, a migration request for migrating financial institution data from a first data environment to a second data environment, the migration request comprising a source file path and a target file location, the first data environment being associated with a private network and the second data environment being associated with a public network;

capturing, by the computer-based system, source file metadata based at least in part on the source file path and the migration request;

converting, by the computer-based system, a source file associated with the migration request to a parquet file using a parquet conversion function;

identifying, by the computer-based system, personally identifiable information comprising at least a transaction account member number in the parquet file; and

in response to receiving an approval to migrate the financial institution data from the first data environment to the second data environment from a user associated with the personally identifiable information:

transferring, by the computer-based system, the converted source file from the first data environment to an intermediate data environment via a first transfer process, the intermediate data environment being associated with the private network; and

transferring, by the computer-based system, the converted source file from the intermediate data environment to the second data environment via a second transfer process.

2. The method of claim 1 , further comprising generating, by the computer-based system, a native agent instruction set based at least in part on the source file metadata and a predefined template associated with the first data environment.

3. The method of claim 2 , further comprising conducting, by the computer-based system, a schedule of operations based at least in part on the native agent instruction set via a native agent of the first data environment.

4. The method of claim 1 , further comprising searching, by the computer-based system, the parquet file for a sensitive data element, the sensitive data element comprising the personally identifiable information, and the personally identifiable information being identified in response to searching the parquet file, and generating a message in response to the sensitive data element, the message comprising an approval link for approving a migration of the financial institution data.

5. The method of claim 4 , further comprising receiving, by the computer-based system, an approval message and exiting a hold state.

6. The method of claim 1 , further comprising updating, by the computer-based system, a status dashboard with at least one of a process status and a module status.

7. The method of claim 1 , wherein the source file metadata comprises at least one of a number of columns transferred, user information, one or more timestamps associated with a start or an end of a transfer, or a predicted transfer time.

8. A system comprising:

a processor; and

a tangible, non-transitory memory configured to communicate with the processor, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause the processor to perform operations comprising:

receiving, by the processor, for migrating financial institution data from a first data environment to a second data environment, the migration request comprising a source file path and a target file location, the first data environment being associated with a private network and the second data environment being associated with a public network;

capturing, by the processor, source file metadata based on the source file path and the migration request;

converting, by the processor, a source file associated with the migration request to a parquet file using a parquet conversion function;

identifying, by the processor, personally identifiable information comprising at least a transaction account member number in the parquet file; and

in response to receiving an approval to migrate the financial institution data from the first data environment to the second data environment from a user associated with the personally identifiable information:

transferring, by the processor, the converted source file from the first data environment to an intermediate data environment via the first transfer process; and

transferring, by the processor, the converted source file from the intermediate data environment to the second data environment via a second transfer process.

9. The system of claim 8 , further comprising generating, by the processor, a native agent instruction set based on the source file metadata and a predefined template associated with the first data environment.

10. The system of claim 9 , further comprising conducting, by the processor, a schedule of operations based on the native agent instruction set via a native agent of the first data environment.

11. The system of claim 8 , further comprising searching, by the processor, the parquet file for a sensitive data element, the sensitive data element comprising the personally identifiable information, and the personally identifiable information being identified in response to searching the parquet file, and generating a message in response to the sensitive data element, the message comprising an approval link for approving a migration of the financial institution data.

12. The system of claim 11 , further comprising receiving, by the processor, an approval message and exiting a hold state.

13. The system of claim 8 , further comprising updating, by the processor, a status dashboard with at least one of a process status and a module status.

14. The system of claim 8 , wherein the source file metadata comprises at least one of a number of columns transferred, user information, one or more timestamps associated with a start or an end of a transfer, or a predicted transfer time.

15. An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that, in response to execution by a computer-based system, cause the computer-based system to perform operations comprising:

receiving, by the computer-based system, a migration request for migrating financial institution data from a first data environment to a second data environment, the migration request comprising a source file path and a target file location, the first data environment being associated with a private network and the second data environment being associated with a public network;

capturing, by the computer-based system, source file metadata based on the source file path and the migration request;

converting, by the computer-based system, a source file associated with the migration request to a parquet file using a parquet conversion function;

identifying, by the computer-based system, personally identifiable information comprising at least a transaction account member number in the parquet fil; and

in response to receiving an approval to migrate the financial institution data from the first data environment to the second data environment from a user associated with the personally identifiable information;

transferring, by the computer-based system, the converted source file from the first data environment to an intermediate data environment via a first transfer process, the intermediate data environment being associated with the private network; and

transferring, by the computer-based system, the converted source file from the intermediate data environment to the second data environment via a second transfer process.

16. The article of manufacture of claim 15 , further comprising generating, by the computer-based system, a native agent instruction set based on the source file metadata and a predefined template associated with the first data environment.

17. The article of manufacture of claim 16 , further comprising conducting, by the computer-based system, a schedule of operations based on the native agent instruction set via a native agent of the first data environment.

18. The article of manufacture of claim 15 , further comprising searching, by the computer-based system, the parquet file for a sensitive data element, the sensitive data element comprising the personally identifiable information, and the personally identifiable information being identified in response to searching the parquet file, and generating a message in response to the sensitive data element, the message comprising an approval link for approving a migration of the financial institution data.

19. The article of manufacture of claim 18 , further comprising receiving, by the computer-based system, an approval message, exiting a hold state, and updating a status dashboard with at least one of a process status and a module status.

20. The article of manufacture of claim 15 , wherein the source file metadata comprises at least one of a number of columns transferred, user information, one or more timestamps associated with a start or an end of a transfer, or a predicted transfer time.