Interactive user interface for dynamically updating data and data analysis and query processing
The systems and methods described herein provide highly dynamic and interactive data analysis user interfaces which enable data analysts to quickly and efficiently explore large volume data sources. The data analysis system enables the user to investigate large volumes of data over many different paths of analysis while maintaining detailed and retraceable steps taken by the user over the course of an investigation, as captured via the user's queries and user interaction with the user interfaces provided by the data analysis system. Data analysis paths may involve exploration of high volume data sets, such as Internet proxy data, which may include trillions of rows of data. The data analyst may pursue a data analysis path that involves, among other things, applying filters, joining to other tables in a database, viewing interactive data visualizations, and so on. The data analysis system enables a user to refresh or insert a node in a workflow. The data analysis system further provides data editing, reporting, enrichment, expression language, and charting functionality.
1. A computer-implemented method comprising:
causing simultaneous presentation, in a user interface, of a plurality of user interface boards, wherein:
each of the plurality of user interface boards is represented by a corresponding data node, of a plurality of data nodes, in a data analysis graph;
the plurality of data nodes in the data analysis graph includes: a parent data node, a first data node that is a child of the parent data node, a second data node that is a child of the first data node, and a current data node that is a child of the second data node; and
the current data node is associated with a most recent user interaction among the plurality of user interface boards;
receiving, via the user interface, a request to update a first user interface board that corresponds to the first data node; and
in response to receiving the request to update the first user interface board, automatically:
accessing historical user interaction data associated with the data analysis graph;
identifying, from historical user interaction data associated with the data analysis graph, the current data node of the data analysis graph;
identifying, based on a dependency of the current data node, an active path of the data analysis graph that includes at least: the first data node, the second data node, and the current data node;
identifying from the active path:
an insertion point between the first data node and the parent data node; and
one or more children data nodes of the first data node including at least the second data node and the current data node;
duplicating the one or more children data nodes to generate one or more duplicate children data nodes, wherein each data node of the one or more duplicate children data nodes is a distinct copy of a corresponding data node from the one or more children data nodes including the second data node and the current data node;
generating an updated first data node based on the request to update the first user interface board;
adding a new path to the data analysis graph comprising the updated first data node and the one or more duplicate children data nodes, wherein adding the new path to the data analysis graph comprises:
linking the updated first data node to the parent data node based on the identified insertion point;
linking the one or more duplicate children data nodes to the updated first data node; and
updating the one or more duplicate children data nodes based on the updated first data node; and
causing presentation, in the user interface, of:
a plurality of updated user interface boards corresponding to respective data nodes of the new path of the data analysis graph.
2. The computer-implemented method of claim 1 , wherein the first data node further comprises at least one of: a summary node, a table node, a histogram node, a distribution node, a time series node, a grid node, a heatmap node, a pivot table node, a filter node, a column editor node, an enrich node, a link node, a set math node, or an export node.
3. The computer-implemented method of claim 1 , further comprising:
identifying a non-transformation data node from the active path, wherein identifying the non-transformation data node further comprises:
determining a node type for a data node from the active path; and
comparing the node type to a list of node types that do not perform transformations;
identifying, from the active path, a subset of data nodes that excludes the non-transformation data node;
identifying an abridged data analysis graph as the subset of data nodes; and
causing presentation of the abridged data analysis graph in the user interface.
4. The computer-implemented method of claim 1 , wherein the new path of the data analysis graph comprises at least two linked data nodes according to an order, and wherein causing presentation of the plurality of updated user interface boards includes causing presentation, in the user interface, of at least two user interface boards according to the order.
5. The computer-implemented method of claim 1 , further comprising:
accessing a first data set from a data node in the new path of the data analysis graph;
accessing report configuration data associated with the data node; and
generating, in the user interface, a report comprising data from the first data set based at least in part on the report configuration data.
6. The computer-implemented method of claim 1 , wherein the first data node comprises a set math node, wherein the set math node is configured to at least: filter a first data set, append data from another data set, or remove data based at least on another data set.
7. A non-transitory computer storage medium storing computer-executable instructions that when executed by a computer hardware processor perform operations comprising:
causing simultaneous presentation, in a user interface, of a plurality of user interface boards, wherein:
each of the plurality of user interface boards is represented by a corresponding data node, of a plurality of data nodes, in a data analysis graph;
the plurality of data nodes in the data analysis graph includes: a parent data node, a first data node that is a child of the parent data node, a second data node that is a child of the first data node, and a current data node that is a child of the second data node; and
the current data node is associated with a most recent user interaction among the plurality of user interface boards;
receiving, via the user interface, a request to update a first user interface board that corresponds to the first data node; and
in response to receiving the request to update the first user interface board, automatically:
accessing historical user interaction data associated with the data analysis graph;
identifying, from historical user interaction data associated with the data analysis graph, the current data node of the data analysis graph;
identifying, based on a dependency of the current data node, an active path of the data analysis graph that includes at least: the first data node, the second data node, and the current data node;
identifying from the active path:
an insertion point between the first data node and the parent data node; and
one or more children data nodes of the first data node including at least the second data node and the current data node;
duplicating the one or more children data nodes to generate one or more duplicate children data nodes, wherein each data node of the one or more duplicate children data nodes is a distinct copy of a corresponding data node from the one or more children data nodes including the second data node and the current data node;
generating an updated first data node based on the request to update the first user interface board;
adding a new path to the data analysis graph comprising the updated first data node and the one or more duplicate children data nodes, wherein adding the new path to the data analysis graph comprises:
linking the updated first data node to the parent data node based on the identified insertion point;
linking the one or more duplicate children data nodes to the updated first data node; and
updating the one or more duplicate children data nodes based on the updated first data node; and
causing presentation, in the user interface, of:
a plurality of updated user interface boards corresponding to respective data nodes of the new path of the data analysis graph.
8. The non-transitory computer storage medium of claim 7 , wherein the first data node further comprises at least one of: a summary node, a table node, a histogram node, a distribution node, a time series node, a grid node, a heatmap node, a pivot table node, a filter node, a column editor node, an enrich node, a link node, a set math node, or an export node.
9. The non-transitory computer storage medium of claim 7 , wherein the operations further comprise:
identifying a non-transformation data node from the active path, wherein identifying the non-transformation data node further comprises:
determining a node type for a data node from the active path; and
comparing the node type to a list of node types that do not perform transformations;
identifying, from the active path, a subset of data nodes that excludes the non-transformation data node;
identifying an abridged data analysis graph as the subset of data nodes; and
causing presentation of the abridged data analysis graph in the user interface.
10. The non-transitory computer storage medium of claim 7 , wherein the new path of the data analysis graph comprises at least two linked data nodes according to an order, and wherein causing presentation of the plurality of updated user interface boards includes causing presentation, in the user interface, of at least two user interface boards according to the order.
11. The non-transitory computer storage medium of claim 7 , wherein the operations further comprise:
accessing a first data set from a data node in the new path of the data analysis graph;
accessing report configuration data associated with the data node; and
generating, in the user interface, a report comprising data from the first data set based at least in part on the report configuration data.
12. The non-transitory computer storage medium of claim 7 , wherein the first data node comprises a set math node, wherein the set math node is configured to at least: filter a first data set, append data from another data set, or remove data based at least on another data set.
13. A computer system comprising:
one or more computer processors configured to execute software code to cause the computer system to perform operations comprising:
causing simultaneous presentation, in a user interface, of a plurality of user interface boards, wherein:
each of the plurality of user interface boards is represented by a corresponding data node, of a plurality of data nodes, in a data analysis graph;
the plurality of data nodes in the data analysis graph includes: a parent data node, a first data node that is a child of the parent data node, a second data node that is a child of the first data node, and a current data node that is a child of the second data node; and
the current data node is associated with a most recent user interaction among the plurality of user interface boards;
receiving, via the user interface, a request to update a first user interface board that corresponds to the first data node; and
in response to receiving the request to update the first user interface board, automatically:
accessing historical user interaction data associated with the data analysis graph;
identifying, from historical user interaction data associated with the data analysis graph, the current data node of the data analysis graph;
identifying, based on a dependency of the current data node, an active path of the data analysis graph that includes at least: the first data node, the second data node, and the current data node;
identifying from the active path:
an insertion point between the first data node and the parent data node; and
one or more children data nodes of the first data node including at least the second data node and the current data node;
duplicating the one or more children data nodes to generate one or more duplicate children data nodes, wherein each data node of the one or more duplicate children data nodes is a distinct copy of a corresponding data node from the one or more children data nodes including the second data node and the current data node;
generating an updated first data node based on the request to update the first user interface board;
adding a new path to the data analysis graph comprising the updated first data node and the one or more duplicate children data nodes, wherein adding the new path to the data analysis graph comprises:
linking the updated first data node to the parent data node based on the identified insertion point;
linking the one or more duplicate children data nodes to the updated first data node; and
updating the one or more duplicate children data nodes based on the updated first data node; and
causing presentation, in the user interface, of:
a plurality of updated user interface boards corresponding to respective data nodes of the new path of the data analysis graph.
14. The computer system of claim 13 , wherein the first data node further comprises at least one of: a summary node, a table node, a histogram node, a distribution node, a time series node, a grid node, a heatmap node, a pivot table node, a filter node, a column editor node, an enrich node, a link node, a set math node, or an export node.
15. The computer system of claim 13 , wherein the operations further comprise:
identifying a non-transformation data node from the active path, wherein identifying the non-transformation data node further comprises:
determining a node type for a data node from the active path; and
comparing the node type to a list of node types that do not perform transformations;
identifying, from the active path, a subset of data nodes that excludes the non-transformation data node;
identifying an abridged data analysis graph as the subset of data nodes; and
causing presentation of the abridged data analysis graph in the user interface.
16. The computer system of claim 13 , wherein the new path of the data analysis graph comprises at least two linked data nodes according to an order, and wherein causing presentation of the plurality of updated user interface boards includes causing presentation, in the user interface, of at least two user interface boards according to the order.
17. The computer system of claim 13 , wherein the operations further comprise:
accessing a first data set from a data node in the new path of the data analysis graph;
accessing report configuration data associated with the data node; and
generating, in the user interface, a report comprising data from the first data set based at least in part on the report configuration data.
18. The computer system of claim 13 , wherein the first data node comprises a set math node, wherein the set math node is configured to at least: filter a first data set, append data from another data set, or remove data based at least on another data set.