Cygraph graph data ingest and enrichment pipeline
Methods and systems are described for assessing a computer network using a graph model. In some instances, the methods comprise: receiving data from at least one data stream of a plurality of data streams, wherein the plurality of data streams comprise computer network data provided by one or more data brokers, and wherein the data received from different data streams of the plurality comprise different data formats; converting the data received from the at least one data stream to a common data format comprising a node or an edge; updating a graph model comprising a plurality of nodes and edges stored within a graph database according to the node or edge of the converted data; and providing a user of the computer network with a visualization of a status of the computer network.
1 . A method for assessing a computer network using a graph model comprising:
receiving data from a plurality of data streams, wherein each of the plurality of data streams is received from an associated data broker and comprises computer network data from a plurality of network sensors, and wherein the data received from different data streams of the plurality of data streams comprise different data formats;
converting, using a first conversion module corresponding to a first data format of the different data formats and configured to receive a first data stream of the plurality of data streams, data from the first data stream to a first converted dataset comprising a graph database format of a graph database comprising a first node or a first edge, wherein the data from the first data stream comprises data of the first format received from an associated first plurality of network sensors;
converting, using a second conversion module corresponding to a second data format of the different data formats and configured to receive a second data stream of the plurality of data streams, data from the second data stream to a second converted data set comprising the graph database format of the graph database comprising a second node or a second edge, wherein the data from the second data stream comprises data of the second format received from an associated second plurality of network sensors;
updating a graph model comprising a plurality of nodes and edges stored within the graph database according to at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset, wherein the graph model is updated based on predefined levels of priority associated with a type or a property of the converted data, and wherein the predefined levels of priority govern whether the converted data modifies an existing node or edge of the graph model;
generating an interactive user interface comprising a user-modifiable visualization of the graph model;
receiving a user input via the interactive user interface, wherein the user input comprises a clustering instruction for managing complexity of the visualization of the graph model, the clustering instruction associated with at least one node property; and
managing complexity of the visualization of the graph model based on the clustering instruction by:
identifying a plurality of nodes associated with the at least one node property, and
displaying the plurality of nodes associated with the at least one node property as a single displayed node and displaying a plurality of non-clustered nodes in association with the single displayed node such that relationships between the non-clustered nodes and the plurality of nodes associated with the at least one node property are preserved.
2 . The method of claim 1 , wherein the data received from the plurality of data streams is received on an intermittent basis.
3 . The method of claim 1 , wherein the data received from the plurality of data streams comprises data indicating a present state of the computer network.
4 . The method of claim 1 , wherein the plurality of data streams comprise a network infrastructure data stream, a network information flow data stream, a network vulnerability scan data stream, a network intrusion detection alert data stream, a network mission dependencies data stream, an arbitrary data stream, or any combination thereof.
5 . The method of claim 1 , wherein converting the received data to the graph database format is performed independently of existing graph models stored within the graph database.
6 . The method of claim 1 , wherein a node comprises information about a network device, a network computer, a network machine, a network cyberspace asset, a network domain, a network rule, a network mission objective, a network mission asset, a network mission task, a network alert, a network vulnerability state, a network vulnerability score, a cyberattack classification, an organization, a user, a geographical area, or any combination thereof.
7 . The method of claim 1 , wherein the graph database format for at least one of the first node and the second node comprises a unique identifier (UID), a category description, a display name description, one or more key-value property fields, and an aggregation field.
8 . The method of claim 7 , wherein the aggregation field comprises instructions for modifying the one or more key-value property fields of a corresponding node within the graph model when at least one of the first node and the second node is integrated with the graph model.
9 . The method of claim 1 , wherein an edge comprises information about a relationship between a starting node and a destination node.
10 . The method of claim 1 , wherein the graph database format for an edge comprises a unique identifier (UID), a category description, a display name description, a starting node UID, a destination node UID, one or more key-value property fields, and an aggregation field.
11 . The method of claim 1 , wherein updating the graph model comprises adding at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset.
12 . The method of claim 1 , wherein updating the graph model comprises deleting a node or edge of the plurality of nodes and edges stored within the graph model that corresponds to at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset.
13 . The method of claim 1 , wherein updating the graph model comprises modifying a property associated with a node or edge of the plurality of nodes and edges stored within the graph model according to that of at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset.
14 . The method of claim 1 , wherein the graph model is updated based on a precedence table, wherein the precedence table defines levels of priority associated with a type or a property of the first converted dataset and the second converted dataset.
15 . The method of claim 1 , wherein updating the graph model comprises:
comparing a node type of the first node to a node type of a corresponding node in the graph model;
determining a relative importance between the node type of the first node and the node type of the corresponding node; and
modifying the corresponding node based on the relative importance of the node type first node and the node type of the corresponding node.
16 . The method of claim 1 , wherein the clustering instruction comprises a selection of the least one node property and identifying the plurality of nodes associated with the at least one node property comprises identifying a plurality of nodes comprising the at least one node property selected by the user.
17 . A computing system for assessing a computer network using a graph database, comprising:
one or more data input devices configured to receive data from a plurality of data streams;
one or more processors;
memory; and
one or more programs stored in the memory that, when executed by the one or more processors, cause the one or more processors to perform a method comprising:
receiving the data from the plurality of data streams, wherein each of the plurality of data streams is received from an associated data broker and comprises computer network data from a plurality of network sensors, and wherein the data received from different data streams of the plurality of data streams comprise a plurality of different data formats;
converting, using a first conversion module corresponding to a first data format of the different data formats and configured to receive a first data stream of the plurality of data streams, data from the first data stream to a first converted dataset comprising a graph database format comprising a first node or a first edge, wherein the data from the first data stream comprises data of the first format received from an associated first plurality of network sensors;
converting, using a second conversion module corresponding to a second data format of the different data formats and configured to receive a second data stream of the plurality of data streams, data from the second data stream to a second converted dataset comprising the graph database format comprising a second node or a second edge, wherein the data from the second data stream comprises data of the second format received from an associated second plurality of network sensors;
updating a graph model comprising a plurality of nodes and edges stored within the graph database according to at least one of the first node, the second node, the first edge, and second edge of the first converted dataset and the second converted dataset, wherein the graph model is updated based on predefined levels of priority associated with a type or a property of the converted data, and wherein the predefined levels of priority govern whether the converted data modifies an existing node or edge of the graph model;
generating an interactive user interface comprising a user-modifiable visualization of the graph model;
receiving a user input via the interactive user interface, wherein the user input comprises a clustering instruction for managing complexity of the visualization of the graph model, the clustering instruction associated with at least one node property; and
managing complexity of the visualization of the graph model based on the clustering instruction by:
identifying a plurality of nodes associated with the at least one node property, and
displaying the plurality of nodes associated with the at least one node property as a single displayed node and displaying a plurality of non-clustered nodes in association with the single displayed node such that relationships between the non-clustered nodes and the plurality of nodes associated with the at least one node property are preserved.
18 . The computing system of claim 17 , wherein the data received from the plurality of data streams is received on an intermittent basis.
19 . The computing system of claim 17 , wherein the data received from the plurality of data streams comprises data indicating a present state of the computer network.
20 . The computing system of claim 17 , wherein the plurality of data streams comprise a network infrastructure data stream, a network information flow data stream, a network vulnerability scan data stream, a network intrusion detection alert data stream, a network mission dependencies data stream, an arbitrary data stream, or any combination thereof.
21 . The computing system of claim 20 , wherein converting the received data to the graph database format is performed independently of existing graph models stored within the graph database.
22 . The computing system of claim 17 , wherein the a node comprises information about a network device, a network computer, a network machine, a network cyberspace asset, a network domain, a network rule, a network mission objective, a network mission asset, a network mission task, a network alert, a network vulnerability state, a network vulnerability score, a cyberattack classification, an organization, a user, a geographical area, or any combination thereof.
23 . The computer system of claim 17 , wherein an edge comprises information about a relationship between a starting node and a destination node.
24 . The computer system of claim 17 , wherein updating the graph model comprises an action selected from the group consisting of adding at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset, deleting a node or edge of the plurality of nodes and edges stored within the graph model that corresponds to at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset, or modifying a property associated with a node or edge of the plurality of nodes and edges stored within the graph model according to that of at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset.
25 . The computing system of claim 17 , wherein the graph model comprises a plurality of layers, each layer associated with a type of computer-network information and comprising a subset of the plurality of nodes and edges in the graph model.
26 . The computing system of claim 25 , wherein the method further comprises sending an alert to the user of the computer network when the graph model has been updated.
27 . The computing system of claim 26 , wherein the method further comprises receiving a domain-specific data query from the user of the computer network.
28 . The computing system of claim 27 , wherein the method further comprises converting the received domain-specific data query to a graph database native query that, when executed upon the graph database, returns matching subgraphs from the plurality of layers of the graph model.
29 . The computing system of claim 28 , wherein the method further comprises providing the user of the computer network with a visualization of the returned matching subgraphs from across the layers of the graph model.
30 . The computing system of claim 17 , wherein the method further comprises allowing the user of the computer network to select a graph dataset from a plurality of graph datasets associated with the graph database.
31 . A non-transitory computer readable storage medium having stored thereon a set of instructions for assessing a computer network using a graph database that, when executed by a computing system, cause the computing system to:
receive data from a plurality of data streams, wherein each of the plurality of data streams received from an associated data broker by and comprises computer network data from a plurality of network sensors, and wherein the data received from different data streams of the plurality of data streams comprise a plurality of different data formats;
convert, using a first conversion module corresponding to a first data format of the different data formats and configured to receive a first data stream of the plurality of data stream, data from the first data stream to a first converted dataset comprising a graph database format comprising a first node or a first edge, wherein the data from the first data stream comprises data of the first format received from an associated first plurality of network sensors;
convert, using a second conversion module corresponding to a second data format of the different data formats and configured to receive a second data stream of the plurality of data streams, data from the second data stream to a second converted dataset comprising the graph database format comprising a second node or a second edge, wherein the data from the second data stream comprises data of the second format received from an associated second plurality of network sensors;
update a graph model comprising a plurality of nodes and edges stored within the graph database according to at least one of the first node, the second node, the first edge, and the second edge of the first converted dataset and the second converted dataset, wherein the graph model is updated based on predefined levels of priority associated with a type or a property of the converted data, and wherein the predefined levels of priority govern whether the converted data modifies an existing node or edge of the graph model;
generate an interactive user interface comprising a user-modifiable visualization of the graph model;
receive a user input via the interactive user interface, wherein the user input comprises a clustering instruction for managing complexity of the visualization of the graph model, the clustering instruction associated with at least one node property; and
managing complexity of the visualization of the graph model based on the clustering instruction by:
identifying a plurality of nodes associated with the at least one node property, and
displaying the plurality of nodes associated with the at least one node property as a single displayed node and displaying a plurality of non-clustered nodes in association with the single displayed node such that relationships between the non-clustered nodes and the plurality of nodes associated with the at least one node property are preserved.