User-interface driven creation and distribution of computer applications

1. A method comprising:

accessing, using one or more processors, a data processing pipeline;

causing presentation of the data processing pipeline on a graphical user interface of a computing device;

receiving a selection of a user interface element within the graphical user interface;

in response to receiving the selection, generating a datafile representing a limited run of the data processing pipeline;

generating an executable application associated with the datafile;

submitting the datafile and the executable application to a software framework;

running the executable application using the datafile;

in response to running the executable application, generating datasets that describe a series of data operations of the data processing pipeline, the series of data operations comprising a subset of an entire series of data operations associated with a complete run of the data pipeline;

receiving the datasets;

in response to receiving the datasets, applying the series of data operations to the data processing pipeline;

collecting performance data metrics; and

dynamically updating the graphical user interface with the performance data metrics.

2. The method of claim 1 , further comprising:

receiving, a set of data from a data source;

processing the set of data by applying data transformations on the set of data; and

writing the processed set of data out to a destination.

3. The method of claim 2 , wherein the destination is a target location for the set of data processed by the data processing pipeline.

4. The method of claim 2 , wherein the software framework is a cluster-computing framework.

5. The method of claim 1 , wherein the datable comprises a series of data operations, wherein each operation in the series of data operations is represented as a user interface element on the graphical user interface.

6. The method of claim 1 , wherein the accessed data processing pipeline is configurable by a user.

7. The method of claim 1 , wherein the accessed data processing pipeline is displayed on the graphical user interface of the computing device.

8. The method of claim 5 , wherein the performance data metrics comprise:

a duration time for each operation in the series of operations and an amount of data transferred for each operation in the series of operations.

9. A computing system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

accessing a data processing pipeline;

causing presentation of the data processing pipeline on a graphical user interface of a computing device;

receiving a selection of a user interface element within the graphical user interface;

in response to receiving the selection, generating a datafile representing a limited run of the data processing pipeline;

generating an executable application associated with the datafile;

submitting the datafile and the executable application to a software framework;

running the executable application using the datafile;

in response to running the executable application, generating datasets that describe a series of data operations of the data processing pipeline, the series of data operations comprising a subset of an entire series of data operations associated with a complete run of the data pipeline;

receiving the datasets;

in response to receiving the datasets, applying the series of data operations to the data processing pipeline;

collecting performance data metrics; and

dynamically updating the graphical user interface with the performance data metrics.

10. The computing system of claim 9 , further comprising:

receiving, a set of data from a data source;

processing the set of data by applying data transformations on the set of data; and

writing the processed set of data out to a destination.

11. The computing system of claim 10 , wherein the destinations are target locations for the set of data processed by the data processing pipeline.

12. The computing system of claim 11 , wherein the software framework is a cluster-computing framework.

13. The computing system of claim 9 , wherein the datafile comprises a series of data operations, wherein each operation in the series of data operations is represented as a user interface element on the graphical user interface.

14. The computing system of claim 9 , wherein the accessed data processing pipeline is configurable by a user.

15. The computing system of claim 9 , wherein the accessed data processing pipeline is displayed on the graphical user interface of the computing device.

16. The computing system of claim 13 , wherein the performance data metrics comprise:

a duration time for each operation in the series of operations and an amount of data transferred for each operation in the series of operations.

17. A non-transitory machine storage medium storing instructions that when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing a data processing pipeline;

causing presentation of the data processing pipeline on a graphical user interface of a computing device;

receiving a selection of a first user interface element within the graphical user interface;

in response to receiving the selection, generating a datafile representing a limited run of the data processing pipeline;

generating an executable application associated with the datafile;

submitting the datafile and the executable application to a software framework;

running the executable application using the datable;

in response to running the executable application, generating datasets that describe a series of data operations of the data processing pipeline, the series of data operations comprising a subset of an entire series of data operations associated with a complete run of the data pipeline;

receiving the datasets;

in response to receiving the datasets, applying the series of data operations to the data processing pipeline;

collecting performance data metrics; and

dynamically updating the graphical user interface with the performance data metrics.

18. The non-transitory machine storage medium of claim 17 , further comprising:

receiving, a set of data from a data source;

processing the set of data by applying data transformations on the set of data; and

wilting the processed set of data out to destinations.

19. The non-transitory machine storage medium of claim 18 , wherein the destinations are target locations for the set of data processed by the data processing pipeline.

20. The non-transitory machine storage medium of claim 17 , wherein the software framework is a cluster-computing framework.