System and method for interactive and automated data processing from disparate data sources
A system and method process data transformation between disparate legacy systems of providers by maintaining databases of schema, rules and work flows applicable to the providers and allowing administrators to interactively customize data transformations from unique versions of data sets in one data type and format to other unique versions in other data types and formats in an automated manner, based on schema, rules and workflows. The process is controlled though an interactive user interface.
1. A method of automated data processing between disparate data systems using different schema, comprising:
ingesting into memory, based on input from an administrator, a data set from a legacy database system with a schema that is stored having a first file type;
applying a version identifier to the data set;
interrogating the data set in memory without first migrating the dataset to another database to verify the schema;
transforming the data set into a standardized data set based on a standardized mapping and standardized data type;
interactively presenting an administrator with a work flow editor to enable an administrator to identify and modify rules applicable to the transforming of the data sets;
applying a version number to the standardized data set; and
outputting the data set into additional schema and data types based on interaction with the administrator;
wherein the administrator is alerted to errors in transformation and completed transformations via an interactive user interface.
2. The method of automated data processing according to claim 1 , further comprising:
applying product rules and case rules stored uniquely for data set providers in the database to the transformation of data sets associated with respective providers.
3. The method of automated data processing according to claim 2 , wherein the data type is JSON.
4. The method of automated data processing according to claim 2 , wherein the data type is one of JSON, XML, .csv and pipe delimited.
5. A system for automated data processing between disparate data systems using different schema, comprising:
a processor;
a memory storing an automated data processing engine and an interactive API;
wherein the interactive API is coupled to the automated data processing engine and administrators at entities having disparate data systems;
provider databases coupled to the engine and the API storing schema, data set versions, rules, and workflows for each provider;
wherein the automated data processing engine is configured to (i) ingest into memory a data set from a legacy database system, based on input from an administrator via the API, with a schema and stored in a first file type; (ii) apply a version identifier to the data set; (iii) interrogate the data set to verify the schema in memory without first migrating the dataset to another database; (iv) transform the data set into a standardized data set based on a standardized mapping and standardized data type; (v) apply a version number to the standardized data set; (vi) output the data set into additional schema and data types based on interaction with the administrator;
wherein the API is configured to interactively present an administrator with a workflow editor to enable the administrator to identify and modify rules applicable to the transforming of the data sets; and
wherein the API is further configured to interact with an administrator to alert the administrator to completed transformations via an interactive user interface.
6. The system according to claim 5 , wherein the automated data processing engine is further configured to apply product rules and case rules stored uniquely for data set providers in the database to the transformation of data sets associated with respective providers.
7. The system according to claim 6 , wherein the data type is JSON.
8. The system according to claim 6 , wherein the data type is one of JSON, XML, .csv and pipe delimited.