Policy-as-code for data assets and remediation in cloud environments
A system and supporting method enable receipt of a computer-coded policy for execution in a control plane associated with a cloud environment to provide data governance in a data plane using one or more data assets of the cloud environment, where the one or more data assets are automatically associated to the computer-coded policy using a set of pre-determined rules associated with the computer-coded policy and using annotations associated with the one or more data assets, and where dynamic changes are to be performed with respect to the annotations based in part on real-time changes to the computer-coded policy to allow monitoring contents of the one or more data assets in accordance with the computer-coded policy and to perform a remediation action that is associated with the one or more data assets in response to a violation associated with the computer-coded policy.
1 . A system comprising:
at least one processor; and
memory storing instructions that when executed by the at least one processor causes the system to:
receive a computer-coded policy for execution in a control plane associated with a cloud environment, the computer-coded policy associated with data governance in a data plane using one or more data assets of the cloud environment;
automatically associate the one or more data assets to the computer-coded policy using a set of pre-determined rules associated with the computer-coded policy and using annotations associated with the one or more data assets, the annotations comprising metadata describing the one or more data assets, the annotations being populated based on one or more parameters provided in an annotation template;
cause one or more dynamic changes to be performed with respect to the annotations based in part on real-time changes to the computer-coded policy;
monitor contents of the one or more data assets in accordance with the computer-coded policy; and
perform a remediation action that is associated with the one or more data assets in response to a violation associated with the computer-coded policy.
2 . The system of claim 1 , wherein the computer-coded policy comprises policy parameters associated with one or more of retention rules, deletion rules, data filtering rules, data disclosure rules, data sovereignty rules, or locality rules.
3 . The system of claim 1 , wherein the memory storing the instructions that when executed by the at least one processor further causes the system to:
provide an interface to enable definition of a policy type and match rules to be part of the computer-coded policy, wherein the match rules are associated with a catalog and with the annotations to enable the system to perform the automatic association of the one or more data assets to the computer-coded policy associate policies; and
determine, using a semantic sub-system, the one or more data assets associated with the policy type and the match rules.
4 . The system of claim 1 , wherein the memory storing the instructions that when executed by the at least one processor further causes the system to:
provide test parameters for the computer-coded policy, the test parameters defining one or more compliance thresholds for the one or more data assets; and
provide a plurality of remediation actions for the one or more data assets, wherein the remediation action performed for the one or more data assets is provided from the plurality of remediation actions based upon the at least one of the one or more compliance thresholds being breached.
5 . The system of claim 1 , wherein the memory storing the instructions that when executed by the at least one processor further causes the system to:
generate instructions based in part on the execution of the computer-coded policy in the control plane; and
perform a deletion or an addition in a retention table, based in part on the instructions, to enforce the remediation action that is associated with the one or more data assets.
6 . The system of claim 1 , wherein the remediation action is one of altering access controls to a data store of the one or more data assets or performing soft-deletes or hard-deletes to expunge non-compliant data of the one or more data assets.
7 . A computer-implemented method comprising:
receiving a computer-coded policy for data governance using one or more data assets within a cloud environment;
automatically associating the one or more data assets to the computer-coded policy based at least in part on one or more pre-defined capabilities of the one or more data assets and using annotations associated with the one or more data assets, the annotations comprising metadata describing the one or more data assets, the annotations being populated based on one or more parameters provided in an annotation template;
enabling dynamic changes to add or remove a determined data asset of the one or more data assets; and
performing a remediation action that is associated with the one or more data assets in response to a violation associated with the computer-coded policy.
8 . The computer-implemented method of claim 7 , wherein the computer-coded policy comprises policy parameters associated with one or more of retention rules, deletion rules, data filtering rules, data disclosure rules, data sovereignty rules, or locality rules.
9 . The computer-implemented method of claim 7 , further comprising:
providing an interface to enable definition of a policy type and match rules to be part of the computer-coded policy, wherein the match rules are associated with a catalog and with the annotations to perform the automatic association of the one or more data assets to the computer-coded policy associate policies; and
determining, using a semantic sub-system, the one or more data assets associated with the policy type and the match rules.
10 . The computer-implemented method of claim 7 , further comprising:
providing test parameters for the computer-coded policy, the test parameters defining one or more compliance thresholds for the one or more data assets; and
providing a plurality of remediation actions for the one or more data assets, wherein the remediation action performed for the one or more data assets is provided from the plurality of remediation actions based upon the at least one of the one or more compliance thresholds being breached.
11 . The computer-implemented method of claim 7 , further comprising:
generating instructions based in part on execution of the computer-coded policy in a control plane of the cloud environment; and
performing a deletion or an addition in a retention table, based in part on the instructions, to enforce the remediation action that is associated with the one or more data assets.
12 . The computer-implemented method of claim 7 , wherein the remediation action is one of altering access controls to a data store of the one or more data assets or performing soft-deletes or hard-deletes to expunge non-compliant data of the one or more data assets.
13 . The computer-implemented method of claim 7 , further comprising:
enabling, using an interface of a control plane, a preview action associated with the computer-coded policy, wherein the computer-coded policy is applied to a representation of the one or more data assets;
providing results associated with remediation actions or violations for the representation of the one or more data assets; and
allowing publishing of the computer-coded policy to act against the one or more data assets.
14 . The computer-implemented method of claim 7 , further comprising:
enabling one or more of notifications to perform a scheduled or triggered testing of the computer-coded policy against a representation of the one or more data assets, wherein the representation comprises infrastructure built in a test portion of a cloud environment to trigger testing and remediation actions therein and to enable real-time changes to the computer-coded policy.
15 . The computer-implemented method of claim 14 , wherein the triggered testing is based in part on a change in the one or more data assets during performance of a workload.
16 . A non-transitory computer-storage medium storing instructions configured to instruct at least one computing device to:
receive a computer-coded policy for data governance using one or more data assets within a cloud environment;
automatically associate the one or more data assets to the computer-coded policy based at least in part on pre-defined capabilities of the one or more data assets and using annotations associated with the one or more data assets, the annotations comprising metadata describing the one or more data assets, the annotations being populated based on one or more parameters provided in an annotation template;
enable dynamic changes to add or remove determined data assets of the one or more data assets; and
perform a remediation action that is associated with the one or more data assets in response to a violation associated with the computer-coded policy.
17 . The non-transitory computer-storage medium of claim 16 , wherein the instructions are configured to instruct at least one computing device further to:
provide an interface to enable definition of a policy type and match rules to be part of the computer-coded policy, wherein the match rules are associated with a catalog and with the annotations to perform the automatic association of the one or more data assets to the computer-coded policy associate policies; and
determine, using a semantic sub-system, the one or more data assets associated with the policy type and the match rules.
18 . The non-transitory computer-storage medium of claim 16 , wherein the instructions are configured to instruct at least one computing device further to:
provide test parameters for the computer-coded policy, the test parameters defining one or more compliance thresholds for the one or more data assets; and
provide a plurality of remediation actions for the one or more data assets, wherein the remediation action performed for the one or more data assets is provided from the plurality of remediation actions based upon the at least one of the one or more compliance thresholds being breached.
19 . The non-transitory computer-storage medium of claim 16 , wherein the instructions are configured to instruct at least one computing device further to:
generate instructions based in part on execution of the computer-coded policy in a control plane; and
perform a deletion or an addition in a retention table, based in part on the instructions, to enforce the remediation action associated with the one or more data assets.
20 . The non-transitory computer-storage medium of claim 16 , wherein the instructions are configured to instruct at least one computing device further to:
enable, using an interface of a control plane, a preview action associated with the computer-coded policy, wherein the computer-coded policy is applied to a representation of the one or more data assets;
provide results associated with remediation actions or violations for the representation of the one or more data assets; and
allow publishing of the computer-coded policy to act against the one or more data assets.