Learning from historical logs and recommending database operations on a data-asset in an ETL tool
A data analysis server maintains database operation history data and context data for database operations performed on tables by a set of training users. The data analysis server builds predictive models for using the maintained data to recommend database operations and operands to a set of guided users. The data analysis server trains the predictive models by determining and weighting features derived from context data that are predictive of performing database operations to tables with similar context data. Using the predictive model, the data analysis server generates recommended database operations and operands based on context data received from a data analysis application of a guided user and sends the recommendations to the data analysis application for presentation to the guided user.
1. A method executed by one or more computing devices for providing recommendations to a user of an instance of a data analysis application, the method comprising:
profiling, by at least one of the one or more computing devices, context data by capturing database operation history data entries and training context data entries from the context data, the context data comprising entries received from one or more instances of the data analysis application responsive to database operations being performed on tables in the data analysis application;
maintaining, by at least one of the one or more computing devices, for a first set of users, profiled database operation history data and profiled context data for a plurality of database operations performed on a plurality of tables;
generating, by at least one of the one or more computing devices, a plurality of predictive models configured to recommend one or more of: at least one database operation or at least one operand to a second set of users of the data analysis application, each predictive model including a plurality of features corresponding to context data fields from the profiled context data and either a plurality of corresponding to database operations or a plurality of operands for recommendation;
receiving an application log entry comprising application context data, the application log entry received responsive to a user of the second set of users selecting a column in a table in the instance of the data analysis application;
selecting, by at least one of the one or more computing devices, one or more predictive models in the plurality of predictive models based at least in part on the application context data;
generating, by at least one of the one or more computing devices, one or more lists of probabilities by inputting the application context data into the one or more selected predictive models, wherein each list of probabilities comprises a plurality of probability values associated with the plurality of database operations or the plurality of operands;
determining, by at least one of the one or more computing devices, one or more recommendations based at least in part on the one or more lists of probabilities, each recommendation in the one or more recommendations comprising a database operation or an operand; and
transmitting, by at least one of the one or more computing devices, the one or more recommendations to the instance of the data analysis application for presentation to the user.
2. The method of claim 1 , wherein generating the plurality of predictive models comprises, for each predictive model:
determining the plurality of features by selecting the plurality of context data fields from the profiled context data;
determining the plurality of database operations or the plurality of operands for recommendation; and
determining, for each of the plurality of database operations or the plurality of operands, a feature weight for each of the plurality of features, the feature weight corresponding to a measure of predictiveness of the feature with respect to the database operation or operand.
3. The method of claim 1 , wherein the training context data comprises at least one of project metadata, worksheet metadata, and user metadata.
4. The method of claim 1 , wherein at least one predictive model is a multinomial logistic classifier.
5. The method of claim 1 , wherein the application context data comprises at least one of project metadata, worksheet metadata, and user metadata.
6. The method of claim 1 , wherein the one or more recommendations comprise at least one of a join operation and a union operation.
7. The method of claim 1 , wherein the one or more selected predictive models comprise an operation model and an operand model and wherein generating one or more lists of probabilities by inputting the application context data into the one or more selected predictive models comprises:
generating a first list of probabilities by inputting the application context data into the operation model, the first list of probabilities comprising probabilities associated with the plurality of operations; and
generating a second list of probabilities by inputting the application context data and the first list of probabilities into the operand model, the second list of probabilities comprising probabilities associated with the plurality of operands.
8. An apparatus for providing recommendations to a user of an instance of a data analysis application, the apparatus comprising:
one or more processors; and
one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
profile context data by capturing database operation history data entries and training context data entries from the context data, the context data comprising entries received from one or more instances of the data analysis application responsive to database operations being performed on tables in the data analysis application;
maintain, for a first set of users, profiled database operation history data and profiled context data for a plurality of database operations performed on a plurality of tables;
generate a plurality of predictive models configured to recommend one or more of: at least one database operation or at least one operand to a second set of users of the data analysis application, each predictive model including a plurality of features corresponding to context data fields from the profiled context data and either a plurality of corresponding database operations or a plurality of operands for recommendation;
receiving an application log entry comprising application context data, the application log entry received responsive to a user of the second set of users selecting a column in a table in the instance of the data analysis application;
select one or more predictive models in the plurality of predictive models based at least in part on the application context data;
generate one or more lists of probabilities by inputting the application context data into the one or more selected predictive models, wherein each list of probabilities comprises a plurality of probability values associated with the plurality of database operations or the plurality of operands;
determine one or more recommendations based at least in part on the one or more lists of probabilities, each recommendation in the one or more recommendations comprising a database operation or an operand; and
transmit the one or more recommendations to the instance of the data analysis application for presentation to the user.
9. The apparatus of claim 8 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to generate the plurality of predictive models further cause at least one of the one or more processors to, for each predictive model:
determine the plurality of features by selecting the plurality of context data fields from the profiled context data;
determine the plurality of database operations or the plurality of operands for recommendation; and
determine, for each of the plurality of database operations or the plurality of operands, a feature weight for each of the plurality of features, the feature weight corresponding to a measure of predictiveness of the feature with respect to the database operation or operand.
10. The apparatus of claim 8 , wherein the training context data comprises at least one of project metadata, worksheet metadata, and user metadata.
11. The apparatus of claim 8 , wherein at least one predictive model is a multinomial logistic classifier.
12. The apparatus of claim 8 , wherein the application context data comprises at least one of project metadata, worksheet metadata, and user metadata.
13. The apparatus of claim 8 , wherein the one or more recommendations comprise at least one of a join operation and a union operation.
14. The apparatus of claim 8 , wherein the one or more selected predictive models comprise an operation model and an operand model and wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to generate one or more lists of probabilities by inputting the application context data into the one or more selected predictive models further cause at least one of the one or more processors to:
generate a first list of probabilities by inputting the application context data into the operation model, the first list of probabilities comprising probabilities associated with the plurality of operations; and
generate a second list of probabilities by inputting the application context data and the first list of probabilities into the operand model, the second list of probabilities comprising probabilities associated with the plurality of operands.
15. At least one non-transitory computer-readable medium storing computer-readable instructions that, when executed by one or more computing devices, cause at least one of the one or more computing devices to:
profile context data by capturing database operation history data entries and training context data entries from the context data, the context data comprising entries received from one or more instances of the data analysis application responsive to database operations being performed on tables in the data analysis application;
maintain, for a first set of users, profiled database operation history data and profiled context data for a plurality of database operations performed on a plurality of tables;
generate a plurality of predictive models configured to recommend one or more of: at least one database operation or at least one operand to a second set of users of the data analysis application, each predictive model including a plurality of features corresponding to context data fields from the profiled context data and either a plurality of corresponding database operations or a plurality of operands for recommendation;
receiving an application log entry comprising application context data, the application log entry received responsive to a user of the second set of users selecting a column in a table in the instance of the data analysis application;
select one or more predictive models in the plurality of predictive models based at least in part on the application context data;
generate one or more lists of probabilities by inputting the application context data into the one or more selected predictive models, wherein each list of probabilities comprises a plurality of probability values associated with the plurality of database operations or the plurality of operands;
determine one or more recommendations based at least in part on the one or more lists of probabilities, each recommendation in the one or more recommendations comprising a database operation or an operand; and
transmit the one or more recommendations to the instance of the data analysis application for presentation to the user.
16. The at least one non-transitory computer-readable medium of claim 15 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to generate the plurality of predictive models further cause at least one of the one or more computing devices to, for each predictive model:
determine the plurality of features by selecting the plurality of context data fields from the profiled context data;
determine the plurality of database operations or the plurality of operands for recommendation; and
determine, for each of the plurality of database operations or the plurality of operands, a feature weight for each of the plurality of features, the feature weight corresponding to a measure of predictiveness of the feature with respect to the database operation or operand.
17. The at least one non-transitory computer-readable medium of claim 15 , wherein the training context data comprises at least one of project metadata, worksheet metadata, and user metadata.
18. The at least one non-transitory computer-readable medium of claim 15 , wherein at least one predictive model is a multinomial logistic classifier.
19. The at least one non-transitory computer-readable medium of claim 15 , wherein the application context data comprises at least one of project metadata, worksheet metadata, and user metadata.
20. The at least one non-transitory computer-readable medium of claim 15 , wherein the one or more recommendations comprise at least one of a join operation and a union operation.
21. The at least one non-transitory computer-readable medium of claim 15 , wherein the one or more selected predictive models comprise an operation model and an operand model and wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to generate one or more lists of probabilities by inputting the application context data into the one or more selected predictive models further cause at least one of the one or more computing devices to:
generate a first list of probabilities by inputting the application context data into the operation model, the first list of probabilities comprising probabilities associated with the plurality of operations; and
generate a second list of probabilities by inputting the application context data and the first list of probabilities into the operand model, the second list of probabilities comprising probabilities associated with the plurality of operands.