Techniques for visualizing dependencies in a data analytics system
Techniques for visualizing dependencies in a data analytics system are described. In an embodiment, a visualization system may receive datasets and dataset dependency information for the datasets. The visualization system may receive a first type of analysis objects from a first analysis object system and dataset dependency information from an analysis object dependency service. The visualization system may receive a second type of analysis objects from a second analysis object system. The second type of analysis object may include a JSON representation of a set of operations that are applied during analysis such as a set of programming instructions. The visualization system may traverse and analyze the JSON representation to determine, for a given analysis object, one or more parent dependencies. The visualization system can then generate a graph of the datasets and analysis objects.
1. A method comprising:
parsing operations of a first analysis object for visualizing data to identify datasets to be visualized via the operations;
determining, based on the parsing, first dataset dependency information for the first analysis object,
the first dataset dependency information identifying a first set of one or more datasets to which the operations apply and on which the first analysis object depends, each dataset of the first set of one or more datasets contained in one or more data files stored in a storage media,
using the first dataset dependency information to cause a visual graph to be presented in a computer graphical user interface,
the visual graph comprising one or more nodes representing the first set of one or more datasets and a node representing the first analysis object and one or more edges between the one or more nodes representing the first set of one or more datasets and the node representing the first analysis object, each edge of the one or more edges representing a data dependency of the first analysis object on a respective dataset of the first set of one or more datasets,
wherein the method is performed using one or more processors.
2. The method of claim 1 , wherein the parsed operations of the first analysis object comprise PYTHON programming language instructions for analyzing the first set of one or more datasets, a JavaScript Object Notation (JSON) representation of operations to apply to the first set of one or more datasets, or HyperTextMarkupLanguage (HTML) code for generating a document based on the first set of one or more datasets.
3. The method of claim 1 , wherein the node of the presented visual graph representing the first analysis object has a distinct visual appearance in the computer graphical user interface from the one or more nodes of the presented visual graph representing the first set of one or more datasets so as to visually indicate that the node representing the first analysis object represents an analysis object and not a dataset.
4. The method of claim 1 , wherein the parsed operations of the first analysis object comprise programming language instructions for analyzing the first set of one or more datasets selected from the group consisting of MATLAB, R, C, C++, OBJECTIVE-C, JAVA, or SQL programming language instructions.
5. The method of claim 1 , the operations including comments on visualizations of underlying data in the one or more datasets or interval selection of the underlying data in the one or more datasets.
6. The method of claim 5 , wherein the processing the operations of the first analysis object to cause a visualization of the first set of one or more datasets to be presented in the computer graphical user interface is performed in response to detecting a user input directed to the node of the presented visual graph representing the first analysis object.
7. The method of claim 1 , further comprising:
determining second dataset dependency information for the first set of one or more datasets, the second dataset dependency information identifying a second dataset on which the first set of one or more datasets depend, the second dataset contained in one or more data files stored in a storage media; and
using the second dataset dependency information to present in the visual graph a node representing the second dataset and one or more edges between the node representing the second dataset and the one or more nodes representing the first set of one or more datasets, each edge of the one or more edges between the node representing the second dataset and the one or more nodes representing the first set of one or more datasets representing a data dependency of a respective dataset of the first set of one or more datasets on the second dataset.
8. One or more non-transitory storage media storing one or more sequences of instructions which, when executed by one or more processors, cause performance of:
parsing operations of a first analysis object for visualizing data to identify datasets to be visualized via the operations;
determining, based on the parsing, first dataset dependency information for the first analysis object,
the first dataset dependency information identifying a first set of one or more datasets to which the operations apply and on which the first analysis object depends, each dataset of the first set of one or more datasets contained in one or more data files stored in a storage media,
using the first dataset dependency information to cause a visual graph to be presented in a computer graphical user interface, the visual graph comprising one or more nodes representing the first set of one or more datasets and a node representing the first analysis object and one or more edges between the one or more nodes representing the first set of one or more datasets and the node representing the first analysis object, each edge of the one or more edges representing a data dependency of the first analysis object on a respective dataset of the first set of one or more datasets.
9. The one or more non-transitory storage media of claim 8 , wherein the parsed operations of the first analysis object comprise PYTHON programming language instructions for analyzing the first set of one or more datasets, a JavaScript Object Notation (JSON) representation of operations to apply to the first set of one or more datasets, or HyperTextMarkupLanguage (HTML) code for generating a document based on the first set of one or more datasets.
10. The one or more non-transitory storage media of claim 8 , wherein the node of the presented visual graph representing the first analysis object has a distinct visual appearance in the computer graphical user interface from the one or more nodes of the presented visual graph representing the first set of one or more datasets so as to visually indicate that the node representing the first analysis object represents an analysis object and not a dataset.
11. The one or more non-transitory storage media of claim 8 , wherein the parsed operations of the first analysis object comprise programming language instructions for analyzing the first set of one or more datasets selected from the group consisting of MATLAB, R, C, C++, OBJECTIVE-C, JAVA, or SQL programming language instructions.
12. The one or more non-transitory storage media of claim 8 , the operations including comments on visualizations of underlying data in the one or more datasets or interval selection of the underlying data in the one or more datasets.
13. The one or more non-transitory storage media of claim 12 , wherein the processing the operations of the first analysis object to cause a visualization of the first set of one or more datasets to be presented in the computer graphical user interface is performed in response to detecting a user input directed to the node of the presented visual graph representing the first analysis object.
14. The one or more non-transitory storage media of claim 8 , further storing instructions which, when executed by the one or more processors, cause performance of:
determining second dataset dependency information for the first set of one or more datasets, the second dataset dependency information identifying a second dataset on which the first set of one or more datasets depend, the second dataset contained in one or more data files stored in a storage media; and
using the second dataset dependency information to present in the visual graph a node representing the second dataset and one or more edges between the node representing the second dataset and the one or more nodes representing the first set of one or more datasets, each edge of the one or more edges between the node representing the second dataset and the one or more nodes representing the first set of one or more datasets representing a data dependency of a respective dataset of the first set of one or more datasets on the second dataset.
15. A computing system comprising:
one or more processors;
storage media; and
instructions stored in the storage media and configured for execution by the one or more processors to perform:
parsing operations of a first analysis object for visualizing data to identify datasets to be visualized via the operations;
determining, based on the parsing, first dataset dependency information for the first analysis object,
the first dataset dependency information identifying a first set of one or more datasets to which the operations apply and on which the first analysis object depends, each dataset of the first set of one or more datasets contained in one or more data files stored in a storage media,
using the first dataset dependency information to cause a visual graph to be presented in a computer graphical user interface and one or more nodes representing the first set of one or more datasets and a node representing the first analysis object, the visual graph further comprising one or more edges between the one or more nodes representing the first set of one or more datasets and the node representing the first analysis object, each edge of the one or more edges representing a data dependency of the first analysis object on a respective dataset of the first set of one or more datasets.
16. The computing system of claim 15 , wherein the parsed operations of the first analysis object comprise PYTHON programming language instructions for analyzing the first set of one or more datasets, a JavaScript Object Notation (JSON) representation of operations to apply to the first set of one or more datasets, or HyperTextMarkupLanguage (HTML) code for generating a document based on the first set of one or more datasets.
17. The computing system of claim 15 , wherein the node of the presented visual graph representing the first analysis object has a distinct visual appearance in the computer graphical user interface from the one or more nodes of the presented visual graph representing the first set of one or more datasets so as to visually indicate that the node representing the first analysis object represents an analysis object and not a dataset.
18. The computing system of claim 15 , wherein the parsed operations of the first analysis object comprise programming language instructions for analyzing the first set of one or more datasets selected from the group consisting of MATLAB, R, C, C++, OBJECTIVE-C, JAVA, or SQL programming language instructions.
19. The computing system of claim 15 , the operations including comments on visualizations of underlying data in the one or more datasets or interval selection of the underlying data in the one or more datasets.
20. The computing system of claim 15 , further comprising instructions stored in storage media and configured for execution by the one or more processors to perform:
determining second dataset dependency information for the first set of one or more datasets, the second dataset dependency information identifying a second dataset on which the first set of one or more datasets depend, the second dataset contained in one or more data files stored in a storage media; and
using the second dataset dependency information to present in the visual graph a node representing the second dataset and one or more edges between the node representing the second dataset and the one or more nodes representing the first set of one or more datasets, each edge of the one or more edges between the node representing the second dataset and the one or more nodes representing the first set of one or more datasets representing a data dependency of a respective dataset of the first set of one or more datasets on the second dataset.