Data revision control in large-scale data analytic systems

1. A method for data revision control in a large-scale data analytic system:

storing a first version of a first dataset that is derived from a first version of a second dataset based on a first execution of a first version of a driver program;

storing, in a build catalog, a first build catalog entry comprising an identifier of the first version of the first dataset, an identifier of the first version of the second dataset, a first branch name, and an identifier of the first version of the driver program;

storing a second version of the first dataset that is derived from a second version of the second dataset based on a second execution of the first version of the driver program;

storing, in the build catalog, a second build catalog entry comprising an identifier of the second version of the first dataset, an identifier of the second version of the second dataset, a second branch name that is different from the first branch name, and an identifier of the first version of the driver program, the build catalog including the first build catalog entry and the second build catalog entry; and

causing display of a provenance graph in a graphical user interface based on the first build catalog entry and the second build catalog entry, the provenance graph display comprising: a first node representing the first version of the first dataset, a second node representing the first version of the second dataset, a third node representing the second version of the first dataset, a fourth node, representing the second version of the second dataset, a first directed edge from the first node to the second node, and a second directed edge from the third node to the fourth node.

2. The method of claim 1 , further comprising:

storing a first transaction entry in a database, the first transaction entry comprising a first transaction commit identifier of the first version of the first dataset;

wherein the first build catalog entry comprises the first transaction commit identifier;

storing a second transaction entry in the database, the second transaction entry comprising a second transaction commit identifier of the first version of the second dataset;

wherein the identifier of the first version of the second dataset in the first build catalog entry is the second transaction commit identifier;

storing a third transaction entry in the database, the third transaction entry comprising a third transaction commit identifier of the second version of the second dataset;

wherein the identifier of the second version of the second dataset in the second build catalog entry is the third transaction commit identifier.

3. The method of claim 1 , further comprising storing the first version of the first dataset in a distributed file system.

4. The method of claim 1 , wherein the identifier of the first version of the first dataset is an identifier assigned to a commit of a transaction in context of which the first version of the first dataset is stored.

5. The method of claim 1 , wherein the first version of the driver program, when executed to produce the first version of the first dataset, transforms data of the first version of the second dataset to produce data of the first version of the first dataset.

6. The method of claim 1 , wherein the provenance graph comprises nodes and directed edges there between, each node representing a dataset and each directed edge between two nodes representing a derivation dependency between datasets.

7. The method of claim 1 , wherein a node is color-coded to indicate that the first dataset is out-of-date with respect to the second dataset.

8. One or more non-transitory computer-readable media storing instructions for execution by one or more processors, the instructions configured for performing operations comprising:

storing a first version of a first dataset that is derived from a first version of a second dataset based on a first execution of a first version of a driver program;

storing, in a build catalog, a first build catalog entry comprising an identifier of the first version of the first dataset, an identifier of the first version of the second dataset, a first branch name, and an identifier of the first version of the driver program;

storing a second version of the first dataset that is derived from a second version of the second dataset based on a second execution of the first version of the driver program;

storing, in the build catalog, a second build catalog entry comprising an identifier of the second version of the first dataset, an identifier of the second version of the second dataset, a second branch name that is different from the first branch name, and an identifier of the first version of the driver program, the build catalog including the first build catalog entry and the second build catalog entry; and

causing display of a provenance graph in a graphical user interface based on the first build catalog entry and the second build catalog entry, the provenance graph display comprising: a first node representing the first version of the first dataset, a second node representing the first version of the second dataset, a third node representing the second version of the first dataset, a fourth node, representing the second version of the second dataset, a first directed edge from the first node to the second node, and a second directed edge from the third node to the fourth node.

9. The one or more non-transitory computer-readable media of claim 8 , wherein the operations further comprise:

storing a first transaction entry in a database, the first transaction entry comprising a first transaction commit identifier of the first version of the first dataset;

wherein the first build catalog entry comprises the first transaction commit identifier;

storing a second transaction entry in the database, the second transaction entry comprising a second transaction commit identifier of the first version of the second dataset;

wherein the identifier of the first version of the second dataset in the first build catalog entry is the second transaction commit identifier;

storing a third transaction entry in the database, the third transaction entry comprising a third transaction commit identifier of the second version of the second dataset;

wherein the identifier of the second version of the second dataset in the second build catalog entry is the third transaction commit identifier.

10. The one or more non-transitory computer-readable media of claim 8 , wherein the operations further comprise: storing the first version of the first dataset in a distributed file system.

11. The one or more non-transitory computer-readable media of claim 8 , wherein the identifier of the first version of the first dataset is an identifier assigned to a commit of a transaction in context of which the first version of the first dataset is stored.

12. The one or more non-transitory computer-readable media of claim 8 , wherein the first version of the driver program, when executed to produce the first version of the first dataset, transforms data of the first version of the second dataset to produce data of the first version of the first dataset.

13. The one or more non-transitory computer-readable media of claim 8 , wherein the provenance graph comprises nodes and directed edges there between, each node representing a dataset and each directed edge between two nodes representing a derivation dependency between datasets.

14. The one or more non-transitory computer-readable media of claim 8 , wherein a node is color-coded to indicate that the first dataset is out-of-date with respect to the second dataset.

15. A system, comprising:

one or more processors;

one or more storage media storing one or more programs which, when executed by the one or more processors, cause:

storing a first version of a first dataset that is derived from a first version of a second dataset based on a first execution of a first version of a driver program;

storing, in a build catalog, a first build catalog entry comprising an identifier of the first version of the first dataset, an identifier of the first version of the second dataset, a first branch name, and an identifier of the first version of the driver program;

storing a second version of the first dataset that is derived from a second version of the second dataset based on a second execution of the first version of the driver program;

storing, in the build catalog, a second build catalog entry comprising an identifier of the second version of the first dataset, an identifier of the second version of the second dataset, a second branch name that is different from the first branch name, and an identifier of the first version of the driver program, the build catalog including the first build catalog entry and the second build catalog entry; and

causing display of a provenance graph in a graphical user interface based on the first build catalog entry and the second build catalog entry, the provenance graph display comprising: a first node representing the first version of the first dataset, a second node representing the first version of the second dataset, a third node representing the second version of the first dataset, a fourth node, representing the second version of the second dataset, a first directed edge from the first node to the second node, and a second directed edge from the third node to the fourth node.

16. The system of claim 15 further comprising:

storing a first transaction entry in a database, the first transaction entry comprising a first transaction commit identifier of the first version of the first dataset;

wherein the first build catalog entry comprises the first transaction commit identifier;

storing a second transaction entry in the database, the second transaction entry comprising a second transaction commit identifier of the first version of the second dataset;

wherein the identifier of the first version of the second dataset in the first build catalog entry is the second transaction commit identifier;

storing a third transaction entry in the database, the third transaction entry comprising a third transaction commit identifier of the second version of the second dataset;

wherein the identifier of the second version of the second dataset in the second build catalog entry is the third transaction commit identifier.

17. The system of claim 15 , wherein the first version of the driver program, when executed to produce the first version of the first dataset, transforms data of the first version of the second dataset to produce data of the first version of the first dataset.