Systems and methods for continuous data profiling
The present disclosure is directed to continuous data profiling (CDP). Entities may house large amounts of disorganized and/or duplicative data. To organize and standardize data across a data set, the data may be profiled. However, profiling large data set can be inefficient and give rise to security problems, as profiling datasets typically requires exporting a dataset to a third-party profiling runtime environment. To remedy these issues, the present disclosure is directed to a continuous data profiling platform that comprises a CDP manager communicatively coupled to a client's database. The CDP manager provides access to a CDP API that may install CDP tools on a client's native database environment, enabling the database management system to profile datasets within the client's native database environment, which results in a more efficient use of computing resources and more secure process of profiling datasets.
1. A method of continuously profiling data, the method comprising:
receiving at least one input stream of data;
profiling the at least one input stream of data by,
identifying at least one attribute in the at least one input stream of data, wherein the at least one attribute is associated with a series of features,
determining a profiling score for the at least one attribute based on a source of the at least one input stream of data and an aggregation of the series of features,
determining the at least one attribute is indicative of an address, and
in response to determining the at least one attribute is indicative of the address, processing the at least one attribute through an address library engine that adds the at least one attribute to a library of addresses;
generating a profiled set of data based on the profiling of the at least one input stream of data, wherein the profiled set of data includes the profiling score for the at least one attribute; and
storing the profiled set of data in at least one client database.
2. The method of claim 1 , further comprising:
retrieving at least one set of profiling rules and at least one set of processing rules that correspond to the at least one attribute;
comparing the at least one attribute with the at least one set of profiling rules and the at least one set of processing rules to validate information included in the at least one attribute;
in response to determining that the information included in the at least one attribute is profiled according to the at least one set of processing rules, storing the information for the at least one attribute into at least one profiled format according to the at least one set of processing rules, wherein the information is provisioned via at least one API gateway; and
processing the at least one attribute through at least one set of rules engines.
3. The method of claim 1 , further comprising:
in response to determining that the at least one attribute is indicative of a name, processing the at least one attribute through a name engine that associates the at least one attribute with associated names included in a listing of associated names.
4. The method of claim 1 , further comprising:
receiving at least one set of continuous data profiling tools from a continuous data profiling manager, wherein the continuous data profiling manager is a frontend software application communicatively coupled to the at least one client database.
5. The method of claim 1 , further comprising:
connecting at least one continuous data profiling (CDP) manager application to the at least one client database; and
receiving at least one instruction or at least one function via the at least one continuous data profiling (CDP) manager application.
6. The method of claim 5 ,
wherein the at least one function is a first function for generating a statistical table of profiled data,
wherein the at least one function is a second function for generating a change-data-store table of profiled data,
wherein the at least one function is a third function for managing at least one profiling process, and
wherein the at least one function is a fourth function for managing at least one profile trigger that is compared to the at least one input stream of data.
7. A system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform a process for continuously profiling data, the process comprising:
receiving at least one input stream of data;
profiling the at least one input stream of data by,
identifying at least one attribute in the at least one input stream of data, wherein the at least one attribute is associated with a series of features,
determining a profiling score for the at least one attribute based on a source of the at least one input stream of data and an aggregation of the series of features,
determining the at least one attribute is indicative of an address, and
in response to determining the at least one attribute is indicative of the address, processing the at least one attribute through an address library engine that adds the at least one attribute to a library of addresses;
generating a profiled set of data based on the profiling of the at least one input stream of data, wherein the profiled set of data includes the profiling score for the at least one attribute; and
storing the profiled set of data in at least one client database.
8. The system according to claim 7 , wherein the process further comprises:
retrieving at least one set of profiling rules and at least one set of processing rules that correspond to the at least one attribute;
comparing the at least one attribute with the at least one set of profiling rules and the at least one set of processing rules to validate information included in the at least one attribute;
in response to determining that the information included in the at least one attribute is profiled according to the at least one set of processing rules, storing the information for the at least one attribute into at least one profiled format according to the at least one set of processing rules, wherein the information is provisioned via at least one API gateway; and
processing the at least one attribute through at least one set of rules engines.
9. The system according to claim 7 , wherein the process further comprises:
in response to determining that the at least one attribute is indicative of a name, processing the at least one attribute through a name engine that associates the at least one attribute with associated names included in a listing of associated names.
10. The system according to claim 8 , wherein the process further comprises:
receiving at least one set of continuous data profiling tools from a continuous data profiling manager, wherein the continuous data profiling manager is a frontend software application communicatively coupled to the at least one client database.
11. The system according to claim 7 , wherein the process further comprises:
connecting at least one continuous data profiling (CDP) manager application to the at least one client database; and
receiving at least one instruction or at least one function via the at least one continuous data profiling (CDP) manager application.
12. The system according to claim 11 ,
wherein the at least one function is a first function for generating a statistical table of profiled data,
wherein the at least one function is a second function for generating a change-data-store table of profiled data,
wherein the at least one function is a third function for managing at least one profiling process, and
wherein the at least one function is a fourth function for managing at least one profile trigger that is compared to the at least one input stream of data.
13. A non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for continuously profiling data, the operations comprising:
receiving at least one input stream of data;
profiling the at least one input stream of data by,
identifying at least one attribute in the at least one input stream of data, wherein the at least one attribute is associated with a series of features,
determining a profiling score for the at least one attribute based on a source of the at least one input stream of data and an aggregation of the series of features,
determining the at least one attribute is indicative of an address, and
in response to determining the at least one attribute is indicative of the address, processing the at least one attribute through an address library engine that adds the at least one attribute to a library of addresses;
generating a profiled set of data based on the profiling of the at least one input stream of data, wherein the profiled set of data includes the profiling score for the at least one attribute; and
storing the profiled set of data in at least one client database.
14. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
retrieving at least one set of profiling rules and at least one set of processing rules that correspond to the at least one attribute;
comparing the at least one attribute with the at least one set of profiling rules and the at least one set of processing rules to validate information included in the at least one attribute;
in response to determining that the information included in the at least one attribute is profiled according to the at least one set of processing rules, storing the information for the at least one attribute into at least one profiled format according to the at least one set of processing rules, wherein the information is provisioned via at least one API gateway; and
processing the at least one attribute through at least one set of rules engines.
15. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
in response to determining that the at least one attribute is indicative of a name, processing the at least one attribute through a name engine that associates the at least one attribute with associated names included in a listing of associated names.
16. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
receiving at least one set of continuous data profiling tools from a continuous data profiling manager, wherein the continuous data profiling manager is a frontend software application communicatively coupled to the at least one client database.
17. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
connecting at least one continuous data profiling (CDP) manager application to the at least one client database; and
receiving at least one instruction or at least one function via the at least one continuous data profiling (CDP) manager application,
wherein the at least one function is a first function for generating a statistical table of profiled data,
wherein the at least one function is a second function for generating a change-data-store table of profiled data,
wherein the at least one function is a third function for managing at least one profiling process, and
wherein the at least one function is a fourth function for managing at least one profile trigger that is compared to the at least one input stream of data.