Automated data routing in a data confidence fabric
Routing data in a data confidence fabric. Data ingested into a data confidence fabric is routed to maximize confidence scores and to minimize the amount of missing confidence information. Routing is based on a configuration file and on pathing map information that allows nodes capable of applying the trust insertions set forth in the configuration file to be identified.
1. A method, comprising:
receiving a package at a node of a data confidence fabric, wherein the package includes at least data;
performing a trust insertion on the data based on a configuration file at the node, wherein the configuration file identifies trust insertions that should be applied to the data, wherein the trust insertion improves a confidence score of the data, wherein the confidence score is determined by a scoring equation;
adding an annotation to the package that is associated with the trust insertion at the node;
determining a path for the package in the data confidence fabric such that a likelihood of the trust insertions identified in the configuration file being performed on the data is maximized, wherein the path is determined from the configuration file; and
routing the package using the path.
2. The method of claim 1 , further comprising determining the path by identifying a next node in the path based on a next trust insertion identified from the configuration file and routing the package to the next node, wherein the next node is configured to perform the next trust insertion.
3. The method of claim 2 , further comprising dynamically searching for the next node in the path.
4. The method of claim 3 , wherein dynamically searching includes searching for the next node in the pathing map information stored by the node and/or broadcasting a request to other nodes to identify the next node.
5. The method of claim 1 , further comprising determining a full path for the package in the data confidence fabric by identifying the path that best satisfies trust insertions included in the configuration file, wherein the full path is determined from pathing map information and the configuration file.
6. The method of claim 1 , further comprising determining a path based on a ledger score, wherein a path associated with a ledger that has a best representation of a confidence score rating is selected as the path.
7. The method of claim 1 , wherein performing a trust insertion includes leveraging a nearby node to perform the trust insertion, wherein the nearby node is not part of the data confidence fabric.
8. The method of claim 1 , further comprising determining a path based using a round robin mode, a computation-based mode, or a broadcast mode.
9. The method of claim 1 , further comprising joining a new node to the data confidence fabric, wherein the new node has a confidence score and wherein capabilities of the new node are broadcast to other nodes in the data confidence fabric and stored by the other nodes in their pathing map information that receive the broadcast from the new node.
10. The method of claim 1 , wherein each node includes a routing engine that is configured to determine the path, wherein the routing engine uses the configuration information and pathing map information as inputs to determine the path, wherein the pathing map information includes at least associations between nodes in the data confidence fabric and their trust insertion capabilities.
11. The method of claim 1 , wherein determining the path includes determining the path such that an order of nodes follows an order of trust insertions identified in the configuration file.
12. The method of claim 11 , wherein the order of trust insertions identified in the configuration file are arranged in levels.
13. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
performing a trust insertion on the data based on a configuration file at the node, wherein the configuration file identifies trust insertions that should be applied to the data, wherein the trust insertion improves a confidence score of the data, wherein the confidence score is determined by a scoring equation;
adding an annotation to the package that is associated with the trust insertion at the node;
determining a path for the package in the data confidence fabric such that a likelihood of the trust insertions identified in the configuration file being performed on the data is maximized, wherein the path is determined from the configuration file; and
routing the package using the path.
14. The non-transitory storage medium of claim 13 , the operations further comprising:
determining the path by identifying a next node in the path based on a next trust insertion identified from the configuration file and routing the package to the next node, wherein the next node is configured to perform the next trust insertion; and
dynamically searching for the next node in the path, wherein dynamically searching includes searching for the next node in the pathing map information stored by the node and/or broadcasting a request to other nodes to identify the next node.
15. The non-transitory storage medium of claim 13 , the operations further comprising determining a full path for the package in the data confidence fabric by identifying the path that best satisfies trust insertions included in the configuration file, wherein the full path is determined from pathing map information and the configuration file.
16. The non-transitory storage medium of claim 13 , the operations further comprising determining a path based on a ledger score, wherein a path associated with a ledger that has a best representation of a confidence score rating is selected as the path.
17. The non-transitory storage medium of claim 13 , wherein performing a trust insertion includes leveraging a nearby node to perform the trust insertion, wherein the nearby node is not part of the data confidence fabric.
18. The non-transitory storage medium of claim 13 , the operations further comprising joining a new node to the data confidence fabric, wherein the new node has a confidence score and wherein capabilities of the new node are broadcast to other nodes in the data confidence fabric and stored by the other nodes in their pathing map information that receive the broadcast from the new node.
19. The non-transitory storage medium of claim 13 , wherein each node includes a routing engine that is configured to determine the path, wherein the routing engine uses the configuration information and pathing map information as inputs to determine the path, wherein the pathing map information includes at least associations between nodes in the data confidence fabric and their trust insertion capabilities.
20. The non-transitory storage medium of claim 13 , wherein determining the path includes determining the path such that an order of nodes follows an order of trust insertions identified in the configuration file, wherein the order of trust insertions identified in the configuration file are arranged in levels.