Systems and methods for continuous data profiling
The present disclosure is directed to continuous data profiling (CDP). Entities may house large amounts of disorganized and/or duplicative data. To organize and standardize data across a data set, the data may be profiled. However, profiling large data set can be inefficient and give rise to security problems, as profiling datasets typically requires exporting a dataset to a third-party profiling runtime environment. To remedy these issues, the present disclosure is directed to a continuous data profiling platform that comprises a CDP manager communicatively coupled to a client's database. The CDP manager provides access to a CDP API that may install CDP tools on a client's native database environment, enabling the database management system to profile datasets within the client's native database environment, which results in a more efficient use of computing resources and more secure process of profiling datasets.
1. A system for continuously profiling data, comprising:
a memory configured to store non-transitory computer readable instructions; and
a processor communicatively coupled to the memory, wherein the processor, when executing the non-transitory computer readable instructions, is configured to:
receive a first input stream of data;
profile the first input stream of data by,
identifying at least one attribute in the first input stream of data, wherein the at least one attribute is associated with a series of features, and
determining a profiling score for the at least one attribute based on an aggregation of the series of features;
generate a first profiled set of data based on the profiling of the first input stream of data, wherein the first profiled set of data includes the profiling score for the at least one attribute;
receive a second input steam of data;
compare the second input stream of data to at least one profile trigger;
initiate the at least one profile trigger;
profile the second input stream of data;
generate a second profiled set of data based on the profiling of the second input stream of data; and
store the first profiled set of data and the second profiled set of data in at least one client database.
2. The system of claim 1 , wherein the processor is further configured to:
connect at least one continuous data profiling (CDP) manager application to the at least one client database; and
receive at least one instruction or at least one function via the at least one continuous data profiling (CDP) manager application.
3. The system of claim 2 , wherein the at least one function is a function for generating a statistical table of profiled data.
4. The system of claim 2 , wherein the at least one function is a function for generating a change-data-store table of profiled data.
5. The system of claim 2 , wherein the at least one function is a function for managing at least one profiling process.
6. The system of claim 2 , wherein the at least one function is a function for managing the at least one profile trigger.
7. The system of claim 1 , wherein the at least one profile trigger is at least one of: a temporal trigger, a size-based trigger, and a manual trigger.
8. The system of claim 1 , wherein profiling the first input stream of data comprises the steps of:
retrieve at least one set of profiling rules and at least one set of processing rules that correspond to the at least one attribute;
compare the at least one attribute with the at least one set of profiling rules and the at least one set of processing rules to validate information included in the at least one attribute;
responsive to determining that the information included in the at least one attribute is profiled according to the at least one set of processing rules, store the information for the at least one attribute into at least one profiled format according to the at least one set of processing rules, wherein the information is provisioned via at least one API gateway; and
process the profiled at least one attribute through at least one set of rules engines.
9. The system of claim 8 , wherein processing the profiled at least one attribute through the at least one set of rules engines comprises the step of:
responsive to determining that the at least one attribute is indicative of a name, process the at least one attribute through a name engine that associated the at least one attribute with associated names included in a listing of associated names.
10. The system of claim 8 , wherein processing the profiled at least one attribute through the at least one set of rules engines comprises the step of:
responsive to determining that the at least one attribute is indicative of an address, process the least one attribute through an address library engine that adds the at least one attribute to a library of addresses.
11. The system of claim 3 , wherein the processor is further configured to:
receive at least one query via the at least one continuous data profiling manager application, wherein the at least one query generates the statistical table of profiled data.
12. The system of claim 1 , wherein the processor is further configured to:
receive at least one set of continuous data profiling tools from a continuous data profiling manager, wherein the continuous data profiling manager is a frontend software application communicatively coupled to the at least one client database.
13. A method of continuously profiling data, comprising:
receiving at least one set of continuous data profiling tools in at least one client database;
receiving a first input stream of data in the at least one client database;
profiling the first input stream of data using the at least one set of continuous data profiling tools in the at least one client database by,
identifying at least one attribute in the first input stream of data, wherein the at least one attribute is associated with a series of features, and
determining a profiling score for the at least one attribute based on an aggregation of the series of features;
generate a first profiled set of data based on the profiling of the first input stream of data, wherein the first profiled set of data includes the profiling score for the at least one attribute;
receiving a second input steam of data in the at least one client database;
comparing the second input stream of data to at least one profile trigger stored in the at least one client database;
determining that the at least one profile trigger is triggered;
profiling the second input stream of data using the at least one set of continuous data profiling tools in the at least one client database;
generating a second profiled set of data based on the profiling of the second input stream of data; and
storing the first profiled set of data and the second profiled set of data in at least one client database.
14. The method of claim 13 , wherein the at least one set of continuous data profiling tools is received from a continuous data profiling manager.
15. The method of claim 14 , wherein the continuous data profiling manager is a frontend software application communicatively coupled to the at least one client database.
16. The method of claim 13 , wherein the at least one set of continuous data profiling tools comprises a function for generating a statistical table of profiled data.
17. The method of claim 13 , wherein the at least one set of continuous data profiling tools comprises a function for generating a change-data-store table of profiled data.
18. The method of claim 13 , wherein the at least one set of continuous data profiling tools comprises a function for managing the at least one profile trigger.
19. The method of claim 13 , wherein the at least one profile trigger is at least one of: a temporal trigger, a size-based trigger, and a manual trigger.
20. A non-transitory computer-readable media storing computer executable instructions that when executed cause a computer system to perform a method for continuous data profiling, comprising:
receiving, from a continuous data profiling manager, at least one set of continuous data profiling statistics in at least one client database from an API gateway;
receiving a first input stream of data in the at least one client database;
profiling the first input stream of data using the at least one set of continuous data profiling statistics in the at least one client database by,
identifying at least one attribute in the first input stream of data, wherein the at least one attribute is associated with a series of features, and
determining a profiling score for the at least one attribute based on an aggregation of the series of features;
generate a first profiled set of data based on the profiling of the first input stream of data, wherein the first profiled set of data includes the profiling score for the at least one attribute;
receiving a second input steam of data in the at least one client database;
comparing the second input stream of data to at least one profile trigger stored in the at least one client database;
based on the comparison of the second input stream of data to the at least one profile trigger, determining the at least one profile trigger is triggered;
profiling the second input stream of data using the at least one set of continuous data profiling tools in the at least one client database;
generating a second profiled set of data based on the profiling of the second input stream of data; and
storing the first profiled set of data and the second profiled set of data in at least one client database.