Network anomaly detection
A security system detects anomalous activity in a network. The system logs user activity, which can include ports used, compares users to find similar users, sorts similar users into cohorts, and compares new user activity to logged behavior of the cohort. The comparison can include a divergence calculation. Origins of user activity can also be used to determine anomalous network activity. The hostname, username, IP address, and timestamp can be used to calculate aggregate scores and convoluted scores.
1. A computer system comprising:
one or more non-transitory computer readable storage mediums having program instructions embodied thereon; and
one or more processors configured to execute the program instructions to cause the computer system to:
receive network activity information associated with a plurality of users, wherein the network activity information includes at least indications of resources accessed by the plurality of users;
determine, for each for the resources accessed, and based at least in part on the network activity information, respective inverse user frequency scale factors;
calculate similarity scores for the plurality of users based at least in part on the network activity information and the relevant inverse user frequency scale factors;
sort the plurality of users into a plurality of cohorts based at least in part on the similarity scores;
receive new first network activity information associated with a first user of the plurality of users, wherein the first user is associated with a first cohort of the plurality of cohorts; and
determine, based at least in part on a comparison between at least a portion of the first network activity information and network activity information associated with the first cohort, that the first network activity information associated with the first user is anomalous.
2. The computer system of claim 1 , wherein the one or more processors are further configured to execute the program instructions to further cause the computer system to:
analyze the network activity information to determine any distributed resources among the resources accessed by the plurality of users,
wherein any multiple resources accessed that comprise a single distributed resources are considered a single resource accessed by the plurality of users for the purposes of determining inverse user frequency scale factors, determining similarity scores, and determining that new network activity is anomalous.
3. The computer system of claim 1 , wherein the respective inverse user frequency scale factors are calculated, for each respective resource accessed, by dividing the number of accesses by the number of users, and subtracting the result from one.
4. The computer system of claim 1 , wherein the respective inverse user frequency scale factors are calculated, for each respective resource accessed, by dividing the number of users by the number of accesses, and taking the log of the result.
5. The computer system of claim 1 , the similarity scores are calculated at least in part by determining at least one of: Jaccard similarity scores, or cosine similarity scores.
6. The computer system of claim 1 , wherein the one or more processors are further configured to execute the program instructions to further cause the computer system to:
determine, based at least in part on a comparison between at least a portion of the first network activity information and previous network activity information associated with the first user, that the first network activity information associated with the first user is not anomalous.
7. The computer system of claim 1 , the plurality of users are sorted into the plurality of cohorts further based at least in part on user information associated with the users.
8. A computer-implemented method comprising:
by one or more processors executing program instructions:
receiving network activity information associated with a plurality of users, wherein the network activity information includes at least indications of resources accessed by the plurality of users;
determining, for each for the resources accessed, and based at least in part on the network activity information, respective inverse user frequency scale factors;
calculating similarity scores for the plurality of users based at least in part on the network activity information and the relevant inverse user frequency scale factors;
sorting the plurality of users into a plurality of cohorts based at least in part on the similarity scores;
receiving new first network activity information associated with a first user of the plurality of users, wherein the first user is associated with a first cohort of the plurality of cohorts; and
determining, based at least in part on a comparison between at least a portion of the first network activity information and network activity information associated with the first cohort, that the first network activity information associated with the first user is anomalous.
9. The computer-implemented method of claim 8 further comprising:
by the one or more processors executing program instructions:
analyzing the network activity information to determine any distributed resources among the resources accessed by the plurality of users,
wherein any multiple resources accessed that comprise a single distributed resources are considered a single resource accessed by the plurality of users for the purposes of determining inverse user frequency scale factors, determining similarity scores, and determining that new network activity is anomalous.
10. The computer-implemented method of claim 8 , wherein the respective inverse user frequency scale factors are calculated, for each respective resource accessed, by dividing the number of accesses by the number of users, and subtracting the result from one.
11. The computer-implemented method of claim 8 , wherein the respective inverse user frequency scale factors are calculated, for each respective resource accessed, by dividing the number of users by the number of accesses, and taking the log of the result.
12. The computer-implemented method of claim 8 , the similarity scores are calculated at least in part by determining at least one of: Jaccard similarity scores, or cosine similarity scores.
13. The computer-implemented method of claim 8 further comprising:
by the one or more processors executing program instructions:
determining, based at least in part on a comparison between at least a portion of the first network activity information and previous network activity information associated with the first user, that the first network activity information associated with the first user is not anomalous.
14. The computer-implemented method of claim 8 , the plurality of users are sorted into the plurality of cohorts further based at least in part on user information associated with the users.
15. A non-transitory computer readable storage medium storing computer executable instructions configured for execution by one or more hardware processors of a computer system to cause the computer system to:
receive network activity information associated with a plurality of users, wherein the network activity information includes at least indications of resources accessed by the plurality of users;
determine, for each for the resources accessed, and based at least in part on the network activity information, respective inverse user frequency scale factors;
calculate similarity scores for the plurality of users based at least in part on the network activity information and the relevant inverse user frequency scale factors;
sort the plurality of users into a plurality of cohorts based at least in part on the similarity scores;
receive new first network activity information associated with a first user of the plurality of users, wherein the first user is associated with a first cohort of the plurality of cohorts; and
determine, based at least in part on a comparison between at least a portion of the first network activity information and network activity information associated with the first cohort, that the first network activity information associated with the first user is anomalous.