Data science workflow execution platform with automatically managed code and graph-based data job management
In an embodiment, the disclosure provides a special-purpose computer system executing operations that allow drag-and-drop input to perform data science operations using visual and graphically enabled data analysis, engineering, and modeling. Using the visual techniques of the disclosure, computer systems are programmed to support rapid creation of data science models and automatic generation of output program code, such as dynamically generated Python, to implement a data science project. In this manner, business-savvy users who are not programmers can execute the system to parse through their data and make useful decisions based on the data, with a focus toward results interpretation and away from computer programming.
1 . A computer-implemented data processing method comprising:
using a server computer executing an application layer and a persistent data storage layer, importing, through the application layer, and storing in non-volatile memory, custom, user-specific business logic that is programmed to one or more business data transformation operations;
the importing of the custom, user-specific business logic comprising serializing the custom, user-specific business logic to text with one or more in-memory dependencies detected using a linting program, programmatically calling one or more network calls to send the serialized form of text associated with the custom, user-specific business logic into the application layer; encapsulating the serialized form of text with the dependencies as an instance of a functional representation type, implementing on the instance one or more security controls configured for secure sharing of the instance, and logging the importing as an auditable event; and storing the serialized form of text and any associated metadata in a repository configured as a functioning programming library with install requirements and versioning, wherein the metadata comprises at least a location of the functioning programming library, the location being configured for loading of the functioning programming library;
storing, in at least one of main memory of the server computer or, through the persistent data storage layer, a back-end database, one or more Tiles respectively corresponding to one or more data sources, each Tile being associated with a plurality of metadata including at least permissions metadata, usage metadata, and version metadata;
storing, in main memory, one or more Edges respectively corresponding to one or more data transformation operations, each Edge being associated with a plurality of metadata including at least permissions metadata, usage metadata, and version metadata;
receiving graphical input that graphically arranges, in a graphical user interface, visual representations in the graphical user interface of the one or more Tiles and the one or more Edges into a Mosaic, the Mosaic forming a directed acyclic graph, and storing the Mosaic in non-volatile memory in association with a project name;
programmatically traversing the one or more Tiles and the one or more Edges comprising the Mosaic, including executing the one or more data transformation operations, to produce a final data set, and storing the final data set in non-volatile memory associated with the project name;
at least one of the Edges that is stored in main memory being an Edge Instance of an imported business data transformation operation;
transforming one or more program objects, one or more of the Edges, and the Mosaic into a serialized form of text using a particular serialization method from among a plurality of different serialization methods based on (1) an object type of the one or more program objects, (2) one or more of the Edges, and (3) the Mosaic, including serializing the one or more of the Edges without decorators or annotations and serializing the Mosaic as a containerization configuration file.
2 . The method of claim 1 , the Edge Instance being associated with edge type metadata partly defined by a number of inputs accepted and a number of outputs produced by the Edge Instance.
3 . The method of claim 1 , at least one Edge of the Mosaic comprising a machine learning classification function;
the method further comprising traversing the one or more tiles and the one or more Edges comprising the Mosaic to produce the final data set by dividing model input data into one or more training segments and one or more test segments for executing the machine learning classification function using random sampling, rolling validation, or walk forward validation.
4 . The method of claim 3 , further comprising causing to display, in the graphical user interface, at least one of a validation curve, a learning curve, a calibration curve, or a precision and recall curve related to execution of the machine learning classification function, wherein the machine learning classification function comprises a random forest classifier.
5 . The method of claim 1 , further comprising generating Python program code by:
selecting one of a user-defined Edge comprising a user-defined data transformation operation or a user-defined Edge Queue comprising a plurality of user-defined data transformation operations;
extracting user-defined logic, in the form of Python code, from the selected user-defined Edge or user-defined Edge Queue;
packaging and storing, in memory, the user-defined logic extracted in the form of Python code in a portable and installable Python library.
6 . A computer system, comprising:
one or more processors;
a memory comprising a set of instructions which when executed causes the one or more processors to execute a method, the method comprising:
using a server computer executing an application layer and a persistent data storage layer, importing, through the application layer, and storing in non-volatile memory, custom, user-specific business logic that is programmed to one or more business data transformation operations;
the importing of the custom, user-specific business logic comprising serializing the custom, user-specific business logic to text with one or more in-memory dependencies detected using a linting program, programmatically calling one or more network calls to send the serialized form of text associated with the custom, user-specific business logic into the application layer; encapsulating the serialized form of text with the dependencies as an instance of a functional representation type, implementing on the instance one or more security controls configured for secure sharing of the instance, and logging the importing as an auditable event; and storing the serialized form of text and any associated metadata in a repository configured as a functioning programming library with install requirements and versioning, wherein the metadata comprises at least a location of the functioning programming library, the location being configured for loading of the functioning programming library;
storing, in at least one of main memory of the server computer or, through the persistent data storage layer, a back-end database, one or more Tiles respectively corresponding to one or more data sources, each Tile being associated with a plurality of metadata including at least permissions metadata, usage metadata, and version metadata;
storing, in main memory, one or more Edges respectively corresponding to one or more data transformation operations, each Edge being associated with a plurality of metadata including at least permissions metadata, usage metadata, and version metadata;
receiving graphical input that graphically arranges, in a graphical user interface, visual representations in the graphical user interface of the one or more Tiles and the one or more Edges into a Mosaic, the Mosaic forming a directed acyclic graph, and storing the Mosaic in non-volatile memory in association with a project name;
programmatically traversing the one or more Tiles and the one or more Edges comprising the Mosaic, including executing the one or more data transformation operations, to produce a final data set, and storing the final data set in non-volatile memory associated with the project name;
at least one of the Edges that is stored in main memory being an Edge Instance of an imported business data transformation operation;
transforming one or more program objects, one or more of the Edges, and the Mosaic into a serialized form of text using a particular serialization method from among a plurality of different serialization methods based on (1) an object type of the one or more program objects, (2) one or more of the Edges, and (3) the Mosaic, including serializing the one or more of the Edges without decorators or annotations and serializing the Mosaic as a containerization configuration file.
7 . The computer system of claim 6 , at least one Edge of the Mosaic comprising a machine learning classification function;
the method further comprising traversing the one or more tiles and the one or more Edges comprising the Mosaic to produce the final data set by dividing model input data into one or more training segments and one or more test segments for executing the machine learning classification function using random sampling, rolling validation, or walk forward validation.
8 . The computer system of claim 7 , the method further comprising causing to display, in the graphical user interface, at least one of a validation curve, a learning curve, a calibration curve, or a precision and recall curve related to execution of the machine learning classification function, wherein the machine learning classification function comprises a random forest classifier.
9 . The computer system of claim 6 , the method further comprising generating program code in Python by:
selecting one of a user-defined Edge comprising a user-defined data transformation operation or a user-defined Edge Queue comprising a plurality of user-defined data transformation operations;
extracting user-defined logic, in the form of Python code, from the selected user-defined Edge or user-defined Edge Queue;
packaging and storing, in memory, the user-defined logic extracted in the form of Python code in a portable and installable Python library.
10 . A computer-implemented auto-packaging of user-defined function method, using a server computer, comprising:
automatically extracting a user-defined function written in a development environment;
selecting one of a user defined Edge comprising a user-defined data transformation operation or a user-defined Edge Queue comprising a plurality of user-defined data transformation operations;
serializing the user-defined function along with any dependencies detected using a linting program to a string/textual data;
encapsulating the string/textual data along with any dependencies as an instance of a functional representation;
implementing one or more security controls on the instance, wherein the security controls are configured for secure sharing of the instance;
logging the extracting of the user-defined function as an auditable event;
instantiating an installable package from a portable and installable library based on the serialized form of string/textual data of the user-defined function along with any dependencies;
writing, in the portable and installable library, the serialized form of string/textual data of the user-defined function;
transforming (1) one or more program objects, and (2) the selected user-defined Edge or the selected user-defined Edge Queue into a serialized form of text using a particular serialization method from among a plurality of different serialization methods based on (1) an object type of the one or more program objects, and (2) the selected user-defined Edge or the selected user-defined Edge Queue.
11 . The method of claim 10 , wherein the user-defined function, the installable package, and the portable and installable library comprise PYTHON computer program code.
12 . The method of claim 10 , further comprising automatically packaging the user-defined function in a human-readable language capable of interpretation or compilation, as part of automated library composition.
13 . The method of claim 10 , further comprising automatically packaging the user-defined function in PYTHON as part of automated library composition.
14 . The method of claim 10 , the installable package in the portable and installable library comprising an auto-generated installation file that allows for the installable package to be installed into an arbitrary environment.