PROVIDING FULL DATA PROVENANCE VISUALIZATION FOR VERSIONED DATASETS
Systems and methods for providing full data provenance visualization for versioned datasets. A method includes receiving selection of a versioned dataset that is within a data pipeline system. The method also includes determining the full data provenance of the selected versioned dataset. The full data provenance may comprise a set of versioned datasets. The method further includes providing for display of a visualization of the full data provenance of the selected versioned dataset. The visualization comprises a graph. The graph comprises a compound node for the selected versioned dataset and for each versioned dataset in the set of versioned datasets. The graph further comprises edges connecting the compounds nodes. Each edge represents a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge.
1 . A method, comprising:
at one or more computing devices having one or more processors and memory storing one or more programs executed by the one or more processors to perform the method, performing the operations of:
receiving selection of a versioned dataset that is within a data pipeline system;
determining full data provenance of the selected versioned dataset, the full data provenance comprising a set of versioned datasets;
providing for display of a visualization of the full data provenance of the selected versioned dataset, the visualization comprising a graph, the graph comprising a compound node for the selected versioned dataset and a compound node for each versioned dataset in the set of versioned datasets, the graph further comprising edges connecting the compounds nodes, each edge representing a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge, the graph further comprising one or more compound nodes with a plurality of sub-entries for different versions of a dataset of the one or more compound nodes.
2 . The method of claim 1 , wherein the compound node of the selected versioned dataset indicates a name or identifier of the selected version dataset; and wherein the compound node for each versioned dataset in the set of versioned datasets indicates a name or identifier of the each versioned dataset.
3 . The method of claim 1 , wherein the compound node of the selected versioned dataset comprises a sub-entry representing a particular version of the selected versioned dataset.
4 . The method of claim 1 , wherein the compound node for each versioned dataset in the set of versioned datasets comprises at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset.
5 . The method of claim 1 , wherein a sub-entry of the compound node for a particular versioned dataset in the set of versioned datasets is visually distinguished in the graphical user interface from other compound node sub-entries of the graph to indicate that a version, of the particular versioned dataset, represented by the sub-entry has been flagged in a database as containing invalid data.
6 . The method of claim 1 , wherein an edge in the graph representing a derivation dependency of a first version of a first versioned dataset in the set of versioned datasets on a second version of a second versioned dataset in the set of versioned datasets is visually distinguished from other edges in the graph to indicate that the first version of the first versioned dataset potentially contains invalid data as a result of the derivation dependency.
7 . The method of claim 1 , wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more Spark systems executing a derivation program taking at least one version of another versioned dataset as input to the derivation program.
8 . The method of claim 1 , wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more MapReduce systems executing a derivation program taking at least one version of another versioned dataset as input as input to the derivation program.
9 . One or more non-transitory computer-readable media storing one or more programs, the one or more programs comprising instructions for:
receiving selection of a versioned dataset that is within a data pipeline system;
determining full data provenance of the selected versioned dataset, the full data provenance comprising a set of versioned datasets;
providing for display of a visualization of the full data provenance of the selected versioned dataset, the visualization comprising a graph, the graph comprising a compound node for the selected versioned dataset and a compound node for each versioned dataset in the set of versioned datasets, the graph further comprising edges connecting the compounds nodes, each edge representing a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge, the graph further comprising one or more compound nodes with a plurality of sub-entries for different versions of a dataset of the one or more compound nodes.
10 . The one or more non-transitory computer-readable media of claim 9 , wherein the compound node of the selected versioned dataset indicates a name or identifier of the selected version dataset; and wherein the compound node for each versioned dataset in the set of versioned datasets indicates a name or identifier of the each versioned dataset.
11 . The one or more non-transitory computer-readable media of claim 9 , wherein the compound node of the selected versioned dataset comprises a sub-entry representing a particular version of the selected versioned dataset.
12 . The one or more non-transitory computer-readable media of claim 9 , wherein the compound node for each versioned dataset in the set of versioned datasets comprises at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset.
13 . The one or more non-transitory computer-readable media of claim 9 , wherein a sub-entry of the compound node for a particular versioned dataset in the set of versioned datasets is visually distinguished in the graphical user interface from other compound node sub-entries of the graph to indicate that a version, of the particular versioned dataset, represented by the sub-entry has been flagged in a database as containing invalid data.
14 . The one or more non-transitory computer-readable media of claim 9 , wherein an edge in the graph representing a derivation dependency of a first version of a first versioned dataset in the set of versioned datasets on a second version of a second versioned dataset in the set of versioned datasets is visually distinguished from other edges in the graph to indicate that the first version of the first versioned dataset potentially contains invalid data as a result of the derivation dependency.
15 . The one or more non-transitory computer-readable media of claim 9 , wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more Spark systems executing a derivation program taking at least one version of another versioned dataset as input to the derivation program.
16 . The one or more non-transitory computer-readable media of claim 9 , wherein at least one version of a versioned dataset in the set of versioned datasets contains data generated as a result of one or more MapReduce systems executing a derivation program taking at least one version of another versioned dataset as input as input to the derivation program.
17 . A system comprising:
memory;
one or more processors;
one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for:
receiving selection of a versioned dataset that is within a data pipeline system;
determining full data provenance of the selected versioned dataset, the full data provenance comprising a set of versioned datasets;
providing for display of a visualization of the full data provenance of the selected versioned dataset, the visualization comprising a graph, the graph comprising a compound node for the selected versioned dataset and a compound node for each versioned dataset in the set of versioned datasets, the graph further comprising edges connecting the compounds nodes, each edge representing a derivation dependency between versions of the versioned datasets represented by the compound nodes connected by the edge, the graph further comprising one or more compound nodes with a plurality of sub-entries for different versions of a dataset of the one or more compound nodes.
18 . The system of claim 17 , wherein the compound node of the selected versioned dataset indicates a name or identifier of the selected version dataset; and wherein the compound node for each versioned dataset in the set of versioned datasets indicates a name or identifier of the each versioned dataset.
19 . The system of claim 17 , wherein the compound node of the selected versioned dataset comprises a sub-entry representing a particular version of the selected versioned dataset.
20 . The system of claim 17 , wherein the compound node for each versioned dataset in the set of versioned datasets comprises at least one sub-entry representing a version of the each versioned dataset in the full data provenance of the selected versioned dataset.