Systems and methods for zero-footprint large-scale user-entity behavior modeling
Systems and methods are disclosed herein for reducing storage space used in tracking behavior of a plurality of network endpoints by modeling the behavior with a behavior model. To this end, control circuitry may determine a respective network endpoint, of a plurality of network endpoints, to which each respective record of a plurality of received records corresponds. The control circuitry then may assign a dedicated queue for each respective network endpoint, and transmit, to each dedicated queue, each record that corresponds to the respective network endpoint to which the respective dedicated queue is assigned. The control circuitry may then determine, for each respective network endpoint, a respective behavior model, and may store each respective behavior model to memory.
1. A method for reducing storage space used in tracking behavior of a network endpoint, the method comprising:
determining a record corresponding to a single network flow originating from a network endpoint;
extract a subset of fields from the record;
encoding the subset of fields into a flow word corresponding to the single network flow originating from the network endpoint;
assigning, for the record, a block to a hash table, wherein the block corresponds to the network endpoint;
adding, to a linked list for the block, the flow word corresponding to the single network flow originating from the network endpoint;
generating, for the network endpoint and based on the linked list for the block, an endpoint vector representing a behavior model for the network endpoint, wherein the generating comprises:
concatenating, for the network endpoint, each flow word in the linked list for the block into a string; and
feeding, for the network endpoint, the string into a Document to Vector (doc2vec) algorithm to output the endpoint vector representing the behavior model for the network endpoint;
storing the behavior model in a memory;
determining an anomalous behavior state for the network endpoint by comparing the endpoint vector corresponding to the network endpoint to a normalcy threshold in a multidimensional space; and
tracking behavior of the network endpoint over time by comparing a current position of the endpoint vector corresponding to the network endpoint in the multidimensional space to a previous position of a previous version of the endpoint vector corresponding to the network endpoint in the multidimensional space.
2. The method of claim 1 , wherein the assigning comprises:
detecting that the record corresponds to an unknown network endpoint; and
in response to the detecting, adding the block to the hash table.
3. The method of claim 1 , wherein the generating the behavior model occurs, for the network endpoint, in response to detecting a threshold amount of flow words having been accumulated for the network endpoint.
4. The method of claim 1 , wherein the generating the behavior model comprises:
identifying one or more modules of a plurality of modules that are idle; and
commanding an idle module of the one or more identified idle modules to generate the endpoint vector representing the behavior model.
5. The method of claim 1 , wherein the feeding the string into the doc2vec algorithm comprises analyzing, using the doc2vec algorithm, the string using a shallow neural network.
6. The method of claim 1 , wherein the tracking is performed using a Kalman filter.
7. The method of claim 1 , wherein the tracking further comprises:
tracking the behavior of the network endpoint by deriving a multivariate Gaussian distribution to determine the current position of the endpoint vector corresponding to the network endpoint in the multidimensional space.
8. A system for reducing storage space used in tracking behavior of a network endpoint, the system comprising:
storage circuitry;
communications circuitry; and
control circuitry configured to:
determine a record corresponding to a single network flow originating from a network endpoint;
extract a subset of fields from the record;
encode the subset of fields into a flow word corresponding to the single network flow originating from the network endpoint;
assign, for the record, a block to a hash table, wherein the block corresponds to the network endpoint;
add, to a linked list for the block, the flow word corresponding to the single network flow originating from the network endpoint;
generate, for the network endpoint and based on the linked list for the block, an endpoint vector representing a behavior model for the network endpoint, wherein the generating comprises:
concatenating, for the network endpoint, each flow word in the linked list for the block into a string; and
feeding, for the network endpoint, the string into a Document to Vector (doc2vec) algorithm to output the endpoint vector representing the behavior model for the network endpoint;
store, by the storage circuitry, the behavior model in a memory;
determine an anomalous behavior state for the network endpoint by comparing the endpoint vector corresponding to the network endpoint to a normalcy threshold in a multidimensional space; and
track behavior of the network endpoint over time by comparing a current position of the endpoint vector corresponding to the network endpoint in the multidimensional space to a previous position of a previous version of the endpoint vector corresponding to the network endpoint in the multidimensional space.
9. The system of claim 8 , wherein to assign, the control circuit is further configured to:
detect that the record corresponds to an unknown network endpoint; and
in response to the detection, add the block to the hash table.
10. The system of claim 8 , wherein generating the behavior model occurs, for the network endpoint, in response to detecting a threshold amount of flow words having been accumulated for the network endpoint.
11. The system of claim 8 , wherein to generate the behavior model, the control circuit is further configured to:
identify one or more modules of a plurality of modules that are idle; and
command an idle module of the one or more identified idle modules to generate the endpoint vector representing the behavior model.
12. The system of claim 8 , wherein to feed the string into the doc2vec algorithm, the control circuit is further configured to analyze, using the doc2vec algorithm, the string using a shallow neural network.
13. The system of claim 8 , wherein to track the behavior of the network endpoint, the control circuitry is further configured to track the behavior of the network endpoint over time using a Kalman filter.
14. The system of claim 8 , wherein to track the behavior of the network endpoint, the control circuitry is further configured to:
track the behavior of the network endpoint by deriving a multivariate Gaussian distribution to determine the current position of the endpoint vector corresponding to the network endpoint in the multidimensional space.
15. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations comprising:
determining a record corresponding to a single network flow originating from a network endpoint;
extracting a subset of fields from the record;
encoding the subset of fields into a flow word corresponding to the single network flow originating from the network endpoint;
assigning, for the record, a block to a hash table, wherein the block corresponds to the network endpoint;
adding, to a linked list for the block, the flow word corresponding to the single network flow originating from the network endpoint;
generating, for the network endpoint and based on the linked list for the block, an endpoint vector representing a behavior model for the network endpoint, wherein the generating comprises:
concatenating, for the network endpoint, each flow word in the linked list for the block into a string; and
feeding, for the network endpoint, the string into a Document to Vector (doc2vec) algorithm to output the endpoint vector representing the behavior model for the network endpoint;
storing the behavior model in a memory;
determining an anomalous behavior state for the network endpoint by comparing the endpoint vector corresponding to the network endpoint to a normalcy threshold in a multidimensional space; and
tracking behavior of the network endpoint over time by comparing a current position of the endpoint vector corresponding to the network endpoint in the multidimensional space to a previous position of a previous version of the endpoint vector corresponding to the network endpoint in the multidimensional space.
16. The non-transitory computer-readable medium of claim 15 , wherein the assigning further comprises:
detecting that the record corresponds to an unknown network endpoint; and
in response to detecting, adding the block to the hash table.
17. The non-transitory computer-readable medium of claim 15 , wherein the generating the behavior model occurs, for the network endpoint, in response to detecting a threshold amount of words having been accumulated for the network endpoint.
18. The non-transitory computer-readable medium of claim 15 , wherein the generating the behavior model comprises:
identifying one or more modules of a plurality of modules that are idle; and
commanding an idle module of the one or more identified idle modules to generate the endpoint vector representing the behavior model.
19. The non-transitory computer-readable medium of claim 15 , wherein the feeding the string into the doc2vec algorithm comprises analyzing, using the doc2vec algorithm, the string using a shallow neural network.
20. The non-transitory computer-readable medium of claim 15 , wherein the tracking is performed using a Kalman filter.
21. The non-transitory computer-readable medium of claim 15 , wherein the tracking further comprises:
tracking the behavior of the network endpoint by deriving a multivariate Gaussian distribution to determine the current position of the endpoint vector corresponding to the network endpoint in the multidimensional space.