Data confidence fabric policy-based scoring
Generating policy-based confidence scores for data is disclosed. Data captured by a data confidence fabric is annotated when the data is created, mutated, transited or otherwise handled in the data confidence fabric. The annotations are weighted by a policy to generate policy-based confidence scores. The policy-based confidence scores are used in determining whether the data is sufficiently trusted for use by an application.
1. A method, comprising:
ingesting data into a data confidence fabric;
performing trust insertions on the data as the data traverses the data confidence fabric, wherein each of the trust insertions results in an annotation associated with the data;
generating, by a policy engine, a set of policies, wherein each policy is associated with a separate application, and wherein each policy identifies weights to be applied to the annotations, wherein at least two of the policies have different weights for the same annotations;
selecting, by a scoring engine, a policy within the set of policies from the policy engine based on the application that is to use the data;
applying the policy to the annotations, wherein applying the policy includes applying the weights to the annotations;
generating, by the scoring engine, a confidence score for each trust insertion technology represented in the annotations;
applying, by the scoring engine, weights to the individual confidence scores according to the selected policy; and
generating, by the scoring engine, a policy-based confidence score for the data based on a combination of the individual weighted confidence scores, wherein an application accesses the data associated with the individual weighted confidence scores and generates an output using only data whose policy-based confidence scores exceed a threshold score.
2. The method of claim 1 , wherein the trust insertions include one or more of trusted platform modules, ledgers, immutable storage, public key infrastructure, transport layer security, signatures, and encryption.
3. The method of claim 1 , further comprising storing the data and the annotations in a ledger.
4. The method of claim 1 , further comprising managing the policy with the policy engine, wherein the policy engine facilitates creating the policy, updating the policy, and deleting the policy.
5. The method of claim 1 , further comprising applying the policy, by the scoring engine that stores the policy.
6. The method of claim 5 , further comprising calling the scoring engine to execute the policy stored in a policy engine on the data.
7. The method of claim 6 , further comprising updating the policy a single time for all trust insertions in the data confidence fabric.
8. The method of claim 6 , further comprising applying the policy to all data ingested into the data confidence fabric after the data is ingested in a batch process.
9. The method of claim 6 , further comprising updating the policy-based confidence score after updating the policy.
10. The method of claim 1 , further comprising selecting the policy from the set of policies based on one or more of an execution environment, a host, a class of machine, a context, or combination thereof.
11. The method of claim 10 , wherein the execution environment is one of a production environment or a test environment.
12. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
ingesting data into a data confidence fabric;
performing trust insertions on the data as the data traverses the data confidence fabric, wherein each of the trust insertions results in an annotation associated with the data;
generating, by a policy engine, a set of policies, wherein each policy is associated with a separate application, and wherein each policy identifies weights to be applied to the annotations, wherein at least two of the policies have different weights for the same annotations;
selecting, by a scoring engine, a policy within the set of policies from the policy engine based on the application that is to use the data;
applying the policy to the annotations, wherein applying the policy includes applying the weights to the annotations;
generating, by the scoring engine, a confidence score for each trust insertion technology represented in the annotations;
applying, by the scoring engine, weights to the individual confidence scores according to the selected policy; and
generating, by the scoring engine, a policy-based confidence score for the data based on a combination of the individual weighted confidence scores, wherein an application accesses the data associated with the individual weighted confidence scores and generates an output using only data whose policy-based confidence scores exceed a threshold score.
13. The non-transitory storage medium of claim 12 , wherein the trust insertions include one or more of trusted platform modules, ledgers, immutable storage, public key infrastructure, transport layer security, signatures, and encryption.
14. The non-transitory storage medium of claim 12 , further comprising storing the data and the annotations in a ledger.
15. The non-transitory storage medium of claim 12 , further comprising managing the policy with the policy engine, wherein the policy engine facilitates creating the policy, updating the policy, and deleting the policy.
16. The non-transitory storage medium of claim 12 , further comprising applying the policy, by the scoring engine that stores the policy.
17. The non-transitory storage medium of claim 16 , further comprising calling the scoring engine to execute the policy on the data.
18. The non-transitory storage medium of claim 17 , further comprising updating the policy a single time for all trust insertions in the data confidence fabric.
19. The non-transitory storage medium of claim 17 , further comprising applying the policy to all data ingested into the data confidence fabric after the data is ingested in a batch process.
20. The non-transitory storage medium of claim 12 , further comprising selecting the policy from the set of policies based on one or more of an execution environment, a host, a class of machine, a context, or combination thereof.