Metadata-driven data management platform
The present invention relates, in an embodiment, to a system for automatically ingesting data into a data lake. In an embodiment of the present invention, the system comprises computer readable memory having recorded thereon instructions for execution by a processor having an ingestion engine and a metadata model. In an embodiment of the present invention, the instructions are configured to determine, via the metadata model, a type of metadata the ingested data contains; to generate guidelines for processing and transforming the ingested data based on the determined metadata; to apply the guidelines at the ingestion engine for how the ingestion engine processes and transforms the ingested data based on the determined metadata; and to store the transformed ingested data to a storage repository.
1. A system for automatically ingesting data into a data lake, the system comprising:
computer readable memory having recorded thereon instructions for execution by a processor having an ingestion engine and a metadata model, said instructions configured to:
determine, automatically via the metadata model, a type of metadata the ingested data contains, the ingested data being associated with source metadata following the metadata model, the source metadata comprising a source type of the metadata;
generate guidelines for processing and transforming the ingested data based on the determined metadata;
apply the guidelines at the ingestion engine for how the ingestion engine processes and transforms the ingested data based on the determined metadata; and
store the transformed ingested data to a storage repository based on the determined metadata.
2. The system of claim 1 , wherein the generated guidelines include specifying the source type and encoding for the metadata.
3. The system of claim 1 , wherein the generated guidelines include specifying each type of record within the ingested data and how to manage each type of record.
4. The system of claim 1 , wherein transforming the ingested data includes standardizing the ingested data.
5. The system of claim 1 , comprising a further instruction of applying security policies to the ingested data based on the determined metadata.
6. The system of claim 1 , wherein the determined metadata may be technical metadata or business metadata.
7. The system of claim 1 , wherein the source for the ingested data is selected from the group consisting of a COBOL file, common separated values format, and ORACLE files.
8. A computer-implemented method for automatically ingesting data into a data lake having steps of:
automatically determining a type of metadata the ingested data contains based on a metadata model, the ingested data being associated with source metadata following the metadata model, the source metadata comprising a source type of the metadata;
generating guidelines for processing and transforming the ingested data based on the determined metadata;
processing and transforming the ingested data based on the generated guidelines; and
storing the transformed ingested data to a storage repository based on the determined metadata.
9. The method of claim 8 , wherein the generated guidelines include specifying the source type and encoding for the metadata.
10. The method of claim 8 , wherein the generated guidelines include specifying each type of record within the ingested data and how to manage each type of record.
11. The method of claim 8 , wherein transforming the ingested data includes standardizing the ingested data.
12. The method of claim 8 , further comprising applying security policies to the ingested data based on the determined metadata.
13. The method of claim 8 , wherein the determined metadata may be technical metadata or business metadata.
14. The method of claim 8 , wherein the source for the ingested data is selected from the group consisting of a COBOL file, common separated values format, and ORACLE files.