Systems and methods for generating explainable features for machine learning
A system and method are provided for feature engineering. The method may include obtaining data from a plurality of data sources. The method may also include generating, based on the data, features for training machine learning models using a single data pipeline that uses a feature generation logic that is separate from an execution logic. The method may also include generating, based on relations and/or joins in the feature generation logic, metadata for explaining the features, and generating, based on the metadata, lineage information representing a mapping of one or more features to intermediate tables and/or input tables. The method may also include displaying the lineage information representing the mapping, including allowing a user to select and/or drill down different features, intermediate tables and/or input tables. The method may also include storing the features and the metadata for the features to a feature store.
1 . A method for engineering explainable features for machine learning, the method comprising:
interfacing with a plurality of data sources corresponding to a plurality of cloud data platforms;
obtaining data from the plurality of data sources, wherein the data includes two or more of: streams, files and tables;
identifying updated data sources for triggering feature generation based on profiling a feature generation logic;
determining whether to execute or postpone feature generation based on whether a source of the data is updated; and
in accordance with a determination that the source of the data is updated:
generating, based on the data, a subset of features based on frequency of training of one or more machine learning models using a single data pipeline that uses the feature generation logic that is separate from an execution logic used to generate the features, wherein the feature generation logic is based on a type of data source corresponding to the data, wherein the execution logic is agnostic to the type of data source, wherein the feature generation logic includes relations and/or joins between input tables and/or intermediate tables of the plurality of data sources, wherein features are generated by configuring and executing the single data pipeline based on the execution logic, wherein the feature generation logic is implemented using SQL queries, wherein the feature generation logic includes toggle switches for turning on or off portions specific to data sources of the plurality of data sources, for debugging purposes;
generating, based on the relations and/or joins, metadata for explaining the features for training the one or more machine learning models;
generating, based on the metadata, lineage information representing a mapping of one or more features to intermediate tables and/or input tables; and
combining the metadata with the feature generation logic and storing, to a feature store, the combination in a text-based format for representing structured data, wherein the feature store is used for training the one or more machine learning models, wherein the feature generation logic provides an interface that allows model developers or data scientists to add features to the feature store.
2 . The method of claim 1 , further comprising:
displaying the lineage information representing the mapping of one or more features to intermediate tables and/or input tables, including allowing a user to select and/or drill down different features, intermediate tables and/or input tables.
3 . The method of claim 1 , wherein generating the features comprises configuring and executing queries in the single data pipeline based on the feature generation logic.
4 . The method of claim 1 , further comprising:
while generating the features, generating only a subset of the features based on frequency of training of the one or more machine learning models.