System and method for cloud-based data quality inspection

1 . A device, comprising:

a processing system including a processor; and

a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:

scheduling retrieval of a dataset stored in a cloud network;

determining rules from a preconfigured set of rules that will be used to check a quality of data in the dataset;

determining, based on the preconfigured set of rules, whether any first data quality exceptions associated with the dataset exist;

in response to determining one or more first data quality exceptions associated with the dataset exists:

creating a first exception report from the one or more first data quality exceptions, and

executing a first corrective action on the dataset based on the first exception report; and

in response to determining no first data quality exceptions associated with the dataset exist:

retrieving new data corresponding to changes in the dataset,

determining, based on the preconfigured set of rules, updated rules that will be used to check a quality of the new data,

identifying one or more second data quality exceptions associated with the new data using the updated rules,

creating a second exception report from the one or more second data quality exceptions, and

executing a second corrective action on the new data based on the second exception report.

2 . The device of claim 1 , wherein the dataset comprises a structured query language database.

3 . The device of claim 1 , wherein the preconfigured set of rules is stored in a PySpark config notebook.

4 . The device of claim 3 , wherein the preconfigured set of rules includes missing relation, sequential, unique, limit, category mismatch, value mismatch, parent child category mismatch, regex mismatch, or a combination thereof.

5 . The device of claim 1 , wherein the operations further comprise registering for notifications of changes in the data of the dataset.

6 . The device of claim 5 , wherein the operations further comprise identifying changed data responsive to receiving a notification.

7 . The device of claim 1 , wherein the operations further comprise retrieving second new data corresponding to further changes in the dataset,

determining, based on the preconfigured set of rules, second updated rules that will be used to check a quality of the second new data,

identifying one or more third data quality exceptions associated with the second new data using the second updated rules,

creating a third exception report from the one or more third data quality exceptions, and

executing a third corrective action on the second new data based on the third exception report.

8 . The device of claim 1 , wherein the executing the first corrective action comprises providing a first recommendation for the first corrective action to a user for approval, and

executing the second corrective action comprises providing a second recommendation for the second corrective action to a user for approval.

9 . The device of claim 1 , wherein the processing system comprises a plurality of processors operating in a distributed computing environment.

10 . A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:

scheduling retrieval of a dataset stored in a cloud network;

determining rules from a preconfigured set of rules that will be used to check a quality of data in the dataset;

determining, based on the preconfigured set of rules, whether any first data quality exceptions associated with the dataset exist;

in response to determining one or more first data quality exceptions associated with the dataset exist:

creating a first exception report from the one or more first data quality exceptions, and

executing a first corrective action on the dataset based on the first exception report; and

in response to determining no first data quality exceptions associated with the dataset exist:

retrieving new data corresponding to changes in the dataset,

determining, based on the preconfigured set of rules, updated rules that will be used to check a quality of the new data,

identifying one or more second data quality exceptions associated with the new data using the updated rules,

creating a second exception report from the one or more second data quality exceptions, and

executing a second corrective action on the new data based on the second exception report.

11 . The non-transitory machine-readable medium of claim 10 , wherein the preconfigured set of rules are stored in a PySpark config notebook.

12 . The non-transitory machine-readable medium of claim 11 , wherein the preconfigured set of rules include missing relation, sequential, unique, limit, category mismatch, value mismatch, parent child category mismatch, regex mismatch, or a combination thereof.

13 . The non-transitory machine-readable medium of claim 10 , wherein the operations further comprise registering for notifications of changes in the data of the dataset.

14 . The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise identifying changed data responsive to receiving a notification.

15 . The non-transitory machine-readable medium of claim 14 , wherein the operations further comprise determining which rules from the preconfigured set of rules should be used to identify data quality exceptions in the changed data.

16 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise generating a third exception report from data quality exceptions identified in the changed data.

17 . The non-transitory machine-readable medium of claim 10 , wherein the operations further comprise recommending corrections to the data in the dataset to remove the second data quality exceptions.

18 . The non-transitory machine-readable medium of claim 10 , wherein the processing system comprises a plurality of processors operating in a distributed computing environment.

19 . A method, comprising:

scheduling, by a processing system including a processor, retrieval of a dataset stored in a cloud network;

determining, by the processing system, rules from a preconfigured set of rules that will be used to check a quality of data in the dataset;

determining, by the processing system, based on the preconfigured set of rules, whether any first data quality exceptions associated with the dataset exist;

in response to determining one or more first data quality exceptions associated with the dataset exist:

creating, by the processing system, a first exception report from the one or more first data quality exceptions, and

executing, by the processing system, a first corrective action on the dataset based on the first exception report; and

in response to determining no first data quality exceptions associated with the dataset exist:

retrieving, by the processing system, new data corresponding to changes in the dataset,

determining, by the processing system, based on the preconfigured set of rules, updated rules that will be used to check a quality of the new data,

identifying, by the processing system, one or more second data quality exceptions associated with the new data using the updated rules,

creating, by the processing system, a second exception report from the one or more second data quality exceptions, and

executing, by the processing system, a second corrective action on the new data based on the second exception report.

20 . The method of claim 19 , wherein executing the first corrective action comprises:

identifying, by the processing system, corrections to the dataset using machine learning; and

providing, by the processing system, recommendations for implementing the corrections to a user.