Monitoring an enterprise system utilizing hierarchical clustering of strings in data records
An apparatus includes a processing device configured to obtain data records associated with an enterprise system comprising strings associated with an attribute. The processing device is also configured to generate a similarity matrix with entries comprising values characterizing similarity between respective pairs of the strings. The processing device is further configured to apply a thresholding filter to values in the entries of the similarity matrix to create an adjacency matrix, and to construct a graph network of the data records based at least in part on the adjacency matrix, wherein the graph network comprises edges connecting pairs of the data records. The processing device is further configured to perform a clustering operation on the graph network to identify clusters of the data records for the attribute, and to initiate remedial action in the enterprise system responsive to identifying a given cluster comprising a given subset of the data records.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured to perform steps of:
obtaining a plurality of data records associated with an information technology infrastructure, each of the plurality of data records comprising at least one string associated with at least one attribute;
generating at least one similarity matrix for the strings associated with the at least one attribute, wherein entries of the at least one similarity matrix comprise values characterizing similarity between respective pairs of the strings associated with the at least one attribute;
applying a thresholding filter to values in the entries of the at least one similarity matrix to create at least one adjacency matrix;
constructing at least one graph network of the plurality of data records based at least in part on the at least one adjacency matrix, wherein the at least one graph network comprises edges connecting pairs of the plurality of data records based at least in part on values of entries in the at least one adjacency matrix;
performing at least one clustering operation on the at least one graph network to identify one or more clusters of the plurality of data records for the at least one attribute;
determining a degree of connectivity of a given one of the one or more clusters, the degree of connectivity of the given cluster being determined based at least in part on similarity of string values in two or more of the plurality of data records that are part of the given cluster; and
initiating at least one remedial action on one or more information technology assets of the information technology infrastructure responsive to determining that the degree of connectivity of the given cluster exceeds a designated connectivity threshold.
2. The apparatus of claim 1 wherein generating the at least one similarity matrix comprises performing string similarity calculations for pairs of the strings.
3. The apparatus of claim 2 wherein the string similarity calculations comprise one or more edit distance calculations.
4. The apparatus of claim 3 wherein the one or more edit distance calculations comprises at least one of a Levenshtein edit distance calculation and a Jaro-Winkler edit distance calculation.
5. The apparatus of claim 1 wherein the thresholding filter comprises a shifted Heaviside unit step function.
6. The apparatus of claim 1 wherein applying the thresholding filter comprises setting entries of the at least one similarity matrix with values below a designated threshold to a first value and setting entries of the at least one similarity matrix with values at or above the designated threshold to a second value.
7. The apparatus of claim 6 wherein constructing the at least one graph network comprises connecting pairs of the plurality of data records having entries in the at least one adjacency matrix with the second value, and refraining from connecting pairs of the plurality of data records having entries in the at least one adjacency matrix with the first value.
8. The apparatus of claim 1 wherein performing the at least one clustering operation comprises performing community detection on the at least one graph network to identify the one or more clusters of the plurality of data records for the given attribute, the community detection algorithm comprising a Louvain community detection algorithm.
9. The apparatus of claim 1 wherein the one or more information technology assets of the information technology infrastructure comprise at least one of physical and virtual computing resources in the information technology infrastructure, and wherein initiating the at least one remedial action in the enterprise system comprises at least one of:
applying one or more security hardening procedures to at least one of the one or more information technology assets associated with a given subset of the plurality of data records in the given cluster; and
modifying a configuration of at least one of the one or more information technology assets associated with the given subset of the plurality of data records in the given cluster.
10. The apparatus of claim 1 wherein the plurality of data records are associated with a plurality of users of the information technology infrastructure, and wherein initiating the at least one remedial action comprises at least one of:
blocking access, by one or more of the plurality of users associated with a given subset of the plurality of data records in the given cluster, to at least one of the one or more information technology assets of the information technology infrastructure, the one or more information technology assets of the information technology infrastructure comprising at least one of physical and virtual computing resources; and
monitoring subsequent access, by one or more of the plurality of users associated with the given subset of the plurality of data records in the given cluster, to at least one of the one or more information technology assets of the information technology infrastructure.
11. The apparatus of claim 1 wherein each of the plurality of data records comprises a first string associated with a first attribute and a second string associated with a second attribute.
12. The apparatus of claim 11 wherein:
generating the at least one similarity matrix comprises generating a first similarity matrix for the first strings associated with the first attribute and a second similarity matrix for the second strings associated with the second attribute;
applying the thresholding filter to values in the entries of the at least one similarity matrix to create the at least one adjacency matrix comprises applying a first thresholding filter to values in entries of the first similarity matrix to generate a first adjacency matrix and applying a second thresholding filter to values in entries of the second similarity matrix to generate a second adjacency matrix; and
constructing the at least one graph network of the plurality of data records based at least in part on the at least one adjacency matrix comprises constructing a first graph network based at least in part on the first adjacency matrix and constructing a second graph network based at least in part on the second adjacency matrix.
13. The apparatus of claim 11 wherein:
the at least one graph network comprises a first graph network for the first attribute and a second graph network for the second attribute;
performing the at least one clustering operation on the at least one graph network comprises identifying a first set of one or more clusters of the plurality of data records in the first graph network for the first attribute and identifying a second set of one or more clusters of the plurality of data records in the second graph network for the second attribute; and
initiating the at least one remedial action is further responsive to identifying a given one of the plurality of data records that is in a first cluster with a first subset of the plurality of data records for the first attribute and is in a second cluster with a second subset of the plurality of data records for the second attribute.
14. The apparatus of claim 13 wherein the first attribute comprises a mailing address and the second attribute comprises a name.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform steps of:
obtaining a plurality of data records associated with an information technology infrastructure, each of the plurality of data records comprising at least one string associated with at least one attribute;
generating at least one similarity matrix for the strings associated with the at least one attribute, wherein entries of the at least one similarity matrix comprise values characterizing similarity between respective pairs of the strings associated with the at least one attribute;
applying a thresholding filter to values in the entries of the at least one similarity matrix to create at least one adjacency matrix;
constructing at least one graph network of the plurality of data records based at least in part on the at least one adjacency matrix, wherein the at least one graph network comprises edges connecting pairs of the plurality of data records based at least in part on values of entries in the at least one adjacency matrix;
performing the at least one clustering operation on the at least one graph network to identify one or more clusters of the plurality of data records for the at least one attribute;
determining a degree of connectivity of a given one of the one or more clusters, the degree of connectivity of the given cluster being determined based at least in part on similarity of string values in two or more of the plurality of data records that are part of the given cluster; and
initiating at least one remedial action on one or more information technology assets of the information technology infrastructure responsive to determining that the degree of connectivity of the given cluster exceeds a designated connectivity threshold.
16. The computer program product of claim 15 wherein:
each of the plurality of data records comprises a first string associated with a first attribute and a second string associated with a second attribute;
generating the at least one similarity matrix comprises generating a first similarity matrix for the first strings associated with the first attribute and a second similarity matrix for the second strings associated with the second attribute;
applying the thresholding filter to values in the entries of the at least one similarity matrix to create the at least one adjacency matrix comprises applying a first thresholding filter to values in entries of the first similarity matrix to generate a first adjacency matrix and applying a second thresholding filter to values in entries of the second similarity matrix to generate a second adjacency matrix; and
constructing the at least one graph network of the plurality of data records based at least in part on the at least one adjacency matrix comprises constructing a first graph network based at least in part on the first adjacency matrix and constructing a second graph network based at least in part on the second adjacency matrix.
17. The computer program product of claim 15 wherein:
each of the plurality of data records comprises a first string associated with a first attribute and a second string associated with a second attribute;
the at least one graph network comprises a first graph network for the first attribute and a second graph network for the second attribute;
performing the at least one clustering operation on the at least one graph network comprises identifying a first set of one or more clusters of the plurality of data records in the first graph network for the first attribute and identifying a second set of one or more clusters of the plurality of data records in the second graph network for the second attribute; and
initiating the at least one remedial action is further responsive to identifying a given one of the plurality of data records that is in a first cluster with a first subset of the plurality of data records for the first attribute and is in a second cluster with a second subset of the plurality of data records for the second attribute.
18. A method comprising steps of:
obtaining a plurality of data records associated with an information technology infrastructure, each of the plurality of data records comprising at least one string associated with at least one attribute;
generating at least one similarity matrix for the strings associated with the at least one attribute, wherein entries of the at least one similarity matrix comprise values characterizing similarity between respective pairs of the strings associated with the at least one attribute;
applying a thresholding filter to values in the entries of the at least one similarity matrix to create at least one adjacency matrix;
constructing at least one graph network of the plurality of data records based at least in part on the at least one adjacency matrix, wherein the at least one graph network comprises edges connecting pairs of the plurality of data records based at least in part on values of entries in the at least one adjacency matrix;
performing at least one clustering operation on the at least one graph network to identify one or more clusters of the plurality of data records for the at least one attribute;
determining a degree of connectivity of a given one of the one or more clusters, the degree of connectivity of the given cluster being determined based at least in part on similarity of string values in two or more of the plurality of data records that are part of the given cluster; and
initiating at least one remedial action on one or more information technology assets of the information technology infrastructure responsive to determining that the degree of connectivity of the given cluster exceeds a designated connectivity threshold;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
19. The method of claim 18 wherein:
each of the plurality of data records comprises a first string associated with a first attribute and a second string associated with a second attribute;
generating the at least one similarity matrix comprises generating a first similarity matrix for the first strings associated with the first attribute and a second similarity matrix for the second strings associated with the second attribute;
applying the thresholding filter to values in the entries of the at least one similarity matrix to create the at least one adjacency matrix comprises applying a first thresholding filter to values in entries of the first similarity matrix to generate a first adjacency matrix and applying a second thresholding filter to values in entries of the second similarity matrix to generate a second adjacency matrix; and
constructing the at least one graph network of the plurality of data records based at least in part on the at least one adjacency matrix comprises constructing a first graph network based at least in part on the first adjacency matrix and constructing a second graph network based at least in part on the second adjacency matrix.
20. The method of claim 18 wherein:
each of the plurality of data records comprises a first string associated with a first attribute and a second string associated with a second attribute;
the at least one graph network comprises a first graph network for the first attribute and a second graph network for the second attribute;
performing the at least one clustering operation on the at least one graph network comprises identifying a first set of one or more clusters of the plurality of data records in the first graph network for the first attribute and identifying a second set of one or more clusters of the plurality of data records in the second graph network for the second attribute; and
initiating the at least one remedial action is further responsive to identifying a given one of the plurality of data records that is in a first cluster with a first subset of the plurality of data records for the first attribute and is in a second cluster with a second subset of the plurality of data records for the second attribute.