Data lineage-based anomaly detection
Disclosed are a system and methods for detecting anomalies in a computer network. Files that are stored in the computer network of an organization are enumerated. The locality-sensitive hash values of the enumerated files are calculated. Sensitive files among the enumerated files are identified based on the locality-sensitive hash values of the enumerated files. Similarity between identified sensitive files is determined based on their distance from each other. The identified sensitive files are linked on a timeline based on their similarity and creation time. One or more event filters are applied to the timeline to detect anomalous file events. Files on the timeline that are involved in the anomalous file events and users that operated on the files are flagged. Mitigation is performed on the flagged files and users.
1 . A method of detecting anomalies in a computer network, the method comprising:
enumerating a plurality of files that are on computer systems of the computer network;
calculating a locality-sensitive hash value of each of the plurality of files;
identifying sensitive files from among the plurality of files based on corresponding locality-sensitive hash values of the plurality of files;
determining similarity of the identified sensitive files to each other by comparing corresponding locality-hash values of the identified sensitive files;
linking the identified sensitive files on a timeline based on the similarity of the identified sensitive files to each other and file creation time of the identified sensitive files;
applying one or more event filters to the timeline to detect an anomaly;
flag sensitive files on the timeline that are involved in the anomaly; and
performing mitigation on the flagged sensitive files.
2 . The method of claim 1 , wherein determining the similarity of the identified sensitive files to each other comprises:
determining distance scores between the identified sensitive files; and
comparing the distance scores to similarity thresholds.
3 . The method of claim 1 , wherein the locality-sensitive hash value of each of the plurality of files is calculated using Trend Locality Sensitive Hashing (TLSH) algorithm.
4 . The method of claim 1 , further comprising:
performing mitigation on users that operated on the flagged sensitive files.
5 . The method of claim 1 , wherein the mitigation includes raising an alert.
6 . The method of claim 1 , wherein the anomaly is indicative of intrusion.
7 . A computer system comprising at least one processor and a memory, the memory storing instructions that when executed by the at least one processor cause the computer to:
calculate a locality-sensitive hash value of each of a plurality of files;
identify sensitive files from among the plurality of files based on corresponding locality-sensitive hash values of the plurality of files;
determine similarity of the identified sensitive files to each other by comparing corresponding locality-hash values of the identified sensitive files;
link the identified sensitive files on a timeline based on the similarity of the identified sensitive files to each other and file creation time of the identified sensitive files;
apply one or more event filters to the timeline to detect an anomaly; and
perform mitigation on sensitive files on the timeline that are involved in the anomaly.
8 . The computer system of claim 7 , wherein the similarity of the identified sensitive files to each other is determined by:
determining distance scores between the identified sensitive files; and
comparing the distance scores to similarity thresholds.
9 . The computer system of claim 7 , wherein the locality-sensitive hash value of each of the plurality of files is calculated using Trend Locality Sensitive Hashing (TLSH) algorithm.
10 . The computer system of claim 7 , wherein the instructions in the memory when executed by the at least one processor cause the computer system to:
perform the mitigation on users that operated on the sensitive files that are involved in the anomaly.
11 . The computer system of claim 7 , wherein the mitigation includes raising an alert.
12 . The computer system of claim 7 , wherein the anomaly is indicative of ransomware.