Self-versioning data lake
Embodiments of the present disclosure are directed to maintaining a self-versioning data lake. According to one embodiment, a method for providing access to a data lake can comprise maintaining a corpus of data in an initial version the data lake, monitoring versioning information for each of a plurality of schemas for data submitted to the data lake, and detecting a change in a schema of the plurality of schemas based on the monitoring of the versioning information for the plurality of schema. A new version of the data lake can be created based on the detected changes in the schema while maintaining the initial version of the data lake. A notification of availability of the new version of the data lake can be provided to users of the data lake.
1 . A method for providing a self-versioning data lake, the method comprising:
maintaining, by a host system, a corpus of data in the self-versioning data lake;
performing, by the host system, data quality enforcement on the corpus of data in the self-versioning data lake; and
providing, by the host system, a notification of availability of a new version of the self-versioning data lake to users of the data lake.
2 . The method of claim 1 , wherein maintaining the corpus of data in the self-versioning data lake comprises:
maintaining the corpus of data in an initial version of the self-versioning data lake;
monitoring versioning information for each of a plurality of schemas for data submitted to the self-versioning data lake;
detecting a change in a schema of the plurality of schemas based on the monitoring of the versioning information for the plurality of schema; and
creating the new version of the self-versioning data lake based on the detected changes in the schema while maintaining the initial version of the data lake.
3 . The method of claim 1 , wherein performing data quality enforcement on the corpus of data in the self-versioning data lake comprises:
inspecting data within the corpus of data; and
determining, based on a set of predefined rules for the data, whether the inspected data is suspect.
4 . The method of claim 3 , further comprising, in response to determining the inspected data is suspect:
flagging the suspect data;
correcting the suspect data;
logging the suspect data; and
providing a notification of the suspect data.
5 . The method of claim 4 , further comprising:
determining whether the inspected data comprises regulatory data related to regulatory requirements; and
in response to determining the inspected data comprises regulatory data, validating the regulatory data based on a validation process defined by a client using the regulatory data.
6 . The method of claim 5 , further comprising:
providing access to the flagged, suspect data;
receiving an update to the flagged, suspect data; and
storing the received update to the flagged, suspect data in the corpus of data.
7 . The method of claim 1 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake when the new version of the data lake is created.
8 . The method of claim 1 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake when data to be stored in the data lake is received.
9 . The method of claim 1 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake at scheduled times.
10 . The method of claim 1 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake upon demand.
11 . The method of claim 1 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake on the occurrence of a predefined event.
12 . A system comprising:
a processor; and
a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to:
maintain a corpus of data in a self-versioning data lake;
perform data quality enforcement on the corpus of data in the self-versioning data lake; and
provide a notification of availability of a new version of the self-versioning data lake to users of the data lake.
13 . The system of claim 12 , wherein maintaining the corpus of data in the self-versioning data lake comprises:
maintaining the corpus of data in an initial version of the self-versioning data lake;
monitoring versioning information for each of a plurality of schemas for data submitted to the self-versioning data lake;
detecting a change in a schema of the plurality of schemas based on the monitoring of the versioning information for the plurality of schema; and
creating the new version of the self-versioning data lake based on the detected changes in the schema while maintaining the initial version of the data lake.
14 . The system of claim 13 , wherein performing data quality enforcement on the corpus of data in the self-versioning data lake comprises:
inspecting data within the corpus of data; and
determining, based on a set of predefined rules for the data, whether the inspected data is suspect.
15 . The system of claim 14 , wherein the instructions further cause the processor to, in response to determining the inspected data is suspect:
flag the suspect data;
correct the suspect data;
log the suspect data; and
provide a notification of the suspect data.
16 . The system of claim 15 , wherein the instructions further cause the processor to:
determine whether the inspected data comprises regulatory data related to regulatory requirements; and
in response to determining the inspected data comprises regulatory data, validate the regulatory data based on a validation process defined by a client using the regulatory data.
17 . The system of claim 16 , wherein the instructions further cause the processor to:
provide access to the flagged, suspect data;
receive an update to the flagged, suspect data; and
store the received update to the flagged, suspect data in the corpus of data.
18 . The system of claim 12 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake when the new version of the data lake is created.
19 . The system of claim 12 , wherein data quality enforcement is performed on the corpus of data in the self-versioning data lake when data to be stored in the data lake is received.
20 . A non-transitory, computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to:
maintain a corpus of data in a self-versioning data lake;
perform data quality enforcement on the corpus of data in the self-versioning data lake; and
provide a notification of availability of a new version of the self-versioning data lake to users of the data lake.