Smart data transition to cloud
Examples of systems and method for data transition are described. In an example, the present disclosure provides for automating the process of data movement from on premise to cloud, i.e., source Data warehouse (DWH) movement, ETL to cloud base DWH, ETL. The present disclosure provides for objects identification, metadata extraction, automated data type mapping, target data definition script creation, data extraction in bulk using source native optimized utilities, users and access control mapping to the target DWH, binary object movement, end-end audit report, and reconciliation reports.
1. A system for migrating data from a source data warehouse to a cloud environment, the system comprising:
a processor;
a location recommender coupled to the processor to:
store data type connection information;
predict a location in the cloud environment based on the migrating data; and
predict query performance of the cloud environment and classify tables in the source data warehouse based on query usage, wherein the location recommender predicts the location based on a table accessed by a referential key and wherein the query performance is determined prior to migration to a target data warehouse based on a source database query execution time; and
a data identifier coupled to the processor to:
provide data values based on historical data pertaining to past migration of the cloud, wherein the historical data is to train models for examining the data, executing queries, and responding to the queries, the models being generated based on data and metadata corresponding to the tables;
identify incorrect and sensitive data;
obfuscate the incorrect and the sensitive data; and
load the migrating data on the cloud environment.
2. The system as claimed in claim 1 , wherein the data identifier is to map object compatibility between a source data warehouse and a target data warehouse in the cloud environment.
3. The system as claimed in claim 1 further including a data determiner to:
determine data usage of a source data warehouse at a database level and within a database at a table level;
determine a size of the database; and
reconcile database migration objects based on content and row counts in a source table in the source data warehouse and a migrated table in the cloud environment, wherein the source table stores the data to be migrated and the migrated table is indicative of a table migrated from the source data warehouse to the cloud environment.
4. The system as claimed in claim 1 comprising a report generator to generate an audit report of a source table in the source data warehouse and a target table in the cloud environment, wherein the source table stores the data to be migrated and the target table receives the data.
5. The system as claimed in claim 1 , wherein the location recommender further predicts the location based on one of data usage, frequency of usage, and volume of data.
6. The system as claimed in claim 1 , wherein the query performance is further determined based on one of dependent variables, source database specific variables, and a volume of data.
7. A method for migrating data from a source data warehouse to a cloud environment, the method including:
storing data type connection information;
predicting a location in the cloud environment based on the migrating data;
predicting query performance of the cloud environment and classifying tables in the source data warehouse based on query usage, wherein the location recommender predicts the location based on a table accessed by a referential key and wherein the query performance is determined prior to migration to a target data warehouse based on a source database query execution time;
providing data values based on historical data of past migration of the cloud, wherein the historical data is to train models for examining the data, executing queries, and responding to the queries, the models being generated based on data and metadata corresponding to the tables;
identifying incorrect and sensitive data;
obfuscating the incorrect and the sensitive data; and
loading the migrating data on to the cloud environment.
8. The method as claimed in claim 7 , including mapping object compatibility between a source data warehouse and a target data warehouse in the cloud environment.
9. The method as claimed in claim 7 , including:
determining data usage of a source data warehouse at a database level and within a database at a table level;
determining a size of the database; and
reconciling a database migration object based on content and row counts in a source table in the source data warehouse and a migrated table in the cloud environment, wherein the source table stores the data to be migrated and the migrated table is indicative of a table migrated from the source data warehouse to the cloud environment.
10. The method as claimed in claim 7 , including generating audit reports of a source table in the source data warehouse and a target table in the cloud environment, wherein the source table stores the data to be migrated and the target table receives the data.
11. The method as claimed in claim 7 , including further predicting the location based on one of data usage, frequency of usage, and a volume of data.
12. The method as claimed in claim 7 , wherein the query performance is further determined based on one of dependent variables, source database specific variables, and volume of data.
13. The method as claimed in claim 7 including providing a generic framework for monitoring status of different tasks.
14. The method as claimed in claim 13 wherein the monitoring is performed based on storing logging details and debugging.
15. A non-transitory Computer Readable Medium (CRM) including machine instructions and executable by a processor to migrate data from a source data warehouse to a cloud environment, wherein the instructions are to:
store data type connection information
predict a location in the cloud environment based on the migrating data;
predict query performance of the cloud environment and classify tables in the source data warehouse based on query usage, wherein the location recommender predicts the location based on a table accessed by a referential key and wherein the query performance is determined prior to migration to a target data warehouse based on a source database query execution time;
provide data values based on historical data of past migrations, wherein the historical data is to train models for examining the data, executing queries, and responding to the queries, the models being generated based on data and metadata corresponding to the tables;
identify incorrect and sensitive data;
obfuscate the incorrect and the sensitive data; and
load the migrating data on the cloud environment.
16. The non-transitory CRM as claimed in claim 15 , wherein the instructions are to map object compatibility between a source data warehouse and a destination data warehouse.
17. The non-transitory CRM as claimed in claim 15 , wherein the instructions are to:
determine data usage of a source data warehouse at a database level and within a database at a table level;
determine a size of the database; and
reconcile a database migration object based on content and row counts in a source table in the source data warehouse and a migrated table in the cloud environment, wherein the source table stores the data to be migrated and the migrated table is indicative of a table migrated from the source data warehouse to the cloud environment.
18. The non-transitory CRM as claimed in claim 15 , wherein the instructions are to generate audit reports of a source table in the source data warehouse and a target table in the cloud environment, wherein the source table stores the data to be migrated and the target table receives the data.
19. The non-transitory CRM as claimed in claim 15 , wherein the instructions are to further predict the location based on one of data usage, frequency of usage, and volume of data.
20. The non-transitory CRM as claimed in claim 15 , wherein the query performance is further determined based on one of dependent variables, source database specific variables, and a volume of data.