Random walks to detect dissimilar records
One example method includes identifying dissimilar items in a data set. A data set may be walked one or more times and the nodes or vertices of the data set may be scored based on the number of times the nodes are touched during the walks. Scores below a threshold score are determined to be dissimilar nodes in the data set. This allows a diverse set of nodes to be identified. A dissimilar data set may be used to prevent unintentional bias in algorithmic training.
1. A method, comprising:
performing a collection of walks in a graph starting from a current vertex, wherein the graph represents a data set;
scoring vertices in the graph based on the collection of walks such that each vertex in the graph has a score, wherein the score of each vertex reflects how often each vertex was visited during the collection of walks relative to the other vertices;
constructing a dissimilar set that includes vertices whose scores are below a threshold score;
outputting the dissimilar set that identifies dissimilar vertices in the graph, wherein the current vertex is included in the dissimilar set;
selecting a vertex from a far set, wherein the far set includes all vertices whose score is less than the threshold score; and
performing a collection of walks starting with the selected vertex to generate a second far set, wherein the second far set includes all vertices whose score is below the threshold score for the collection of walks started with the selected vertex.
2. The method of claim 1 , further comprising:
intersecting the far set with the second far set to generate a current far set that includes an intersection of the far set and the second far set;
selecting a vertex from the current far set; and
performing a collection of walks starting from the vertex selected from the current far set.
3. The method of claim 2 , further comprising performing collections of walks from vertices selected from the current far set until the current far set is empty or the dissimilar set includes a predetermined number of vertices.
4. The method of claim 3 , maintaining an audit trail identifying efforts made to find a sufficiently diverse dissimilar set of vertices.
5. The method of claim 1 , further comprising searching a database for the graph.
6. The method of claim 5 , further comprising specifying parameters for searching the database, the parameters including attributes of edges, data type, number of samples to include in the dissimilar set, the threshold value, or combination thereof.
7. The method of claim 1 , further comprising creating a relative ranking for the dissimilar set to measure ethical or bias progress.
8. The method of claim 1 , further comprising assigning a score of 0 to all vertices never touched by the walk.
9. The method of claim 1 , further comprising performing algorithmic training using the dissimilar data set.
10. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
performing a collection of walks in a graph starting from a current vertex, wherein the graph represents a data set;
scoring vertices in the graph based on the collection of walks such that each vertex in the graph has a score, wherein the score of each vertex reflects how often each vertex was visited during the collection of walks relative to the other vertices;
constructing a dissimilar set that includes vertices whose scores are below a threshold score;
outputting the dissimilar set that identifies dissimilar vertices in the graph, wherein the current vertex is included in the dissimilar set;
selecting a vertex from a far set, wherein the far set includes all vertices whose score is less than the threshold score; and
performing a collection of walks starting with the selected vertex to generate a second far set, wherein the second far set includes all vertices whose score is below the threshold score for the collection of walks started with the selected vertex.
11. The non-transitory storage medium of claim 10 , further comprising:
intersecting the far set with the second far set to generate a current far set that includes an intersection of the far set and the second far set;
selecting a vertex from the current far set; and
performing a collection of walks starting from the vertex selected from the current far set.
12. The non-transitory storage medium of claim 11 , further comprising performing collections of walks from vertices selected from the current far set until the current far set is empty or the dissimilar set includes a predetermined number of vertices.
13. The non-transitory storage medium of claim 12 , maintaining an audit trail identifying efforts made to find a sufficiently diverse dissimilar set of vertices.
14. The non-transitory storage medium of claim 10 , further comprising searching a database for the graph.
15. The non-transitory storage medium of claim 14 , further comprising specifying parameters for searching the database, the parameters including attributes of edges, data type, number of samples to include in the dissimilar set, the threshold value, or combination thereof.
16. The non-transitory storage medium of claim 10 , further comprising creating a relative ranking for the dissimilar set to measure ethical or bias progress.
17. The non-transitory storage medium of claim 10 , further comprising assigning a score of 0 to all vertices never touched by the walk.
18. The non-transitory storage medium of claim 10 , further comprising performing algorithmic training using the dissimilar data set.