CENTRALIZED FEATURE MANAGEMENT, MONITORING AND ONBOARDING
The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of features for use by a set of statistical models. Next, the system generates a schema that includes a logical description of data represented by the features and a physical description related to generating and storing the features. The system then outputs the schema for use in managing and sharing the features across the statistical models. Finally, the system updates the outputted schema to reflect one or more parameters from a user.
1 . A method, comprising:
obtaining a set of features for use by a set of statistical models;
generating, by a computer system, a schema comprising:
a logical description of data represented by the features; and
a physical description related to generating and storing the features;
outputting the schema for use in managing and sharing the features across the statistical models; and
updating the outputted schema to reflect one or more parameters from a user.
2 . The method of claim 1 , wherein obtaining the set of features comprises:
obtaining a portion of the schema from one or more users; and
using the portion of the schema to automatically generate the set of features.
3 . The method of claim 1 , further comprising:
monitoring one or more attributes associated with generating and using the features; and
outputting the one or more attributes with the schema.
4 . The method of claim 3 , wherein the one or more attributes comprise at least one of:
a recency;
a usage; and
a distribution.
5 . The method of claim 1 , wherein the logical description and the physical description of the schema comprise:
feature-level attributes that describe a feature in the set of features; and
feature-set-level attributes that describe the set of features.
6 . The method of claim 5 , wherein the feature-set-level attributes in the logical description comprise:
a name;
a category;
a description;
one or more entities;
one or more tags; and
one or more owners.
7 . The method of claim 5 , wherein the feature-level attributes in the logical description comprise:
a name;
a namespace;
a description;
a feature type;
a data type;
an aggregation attribute; and
a transformation option.
8 . The method of claim 5 , wherein the feature-set-level attributes in the physical description comprise:
a location;
a format;
a frequency of generation;
a retention period;
a data availability delay;
a status; and
a source.
9 . The method of claim 5 , wherein the feature-level attributes in the physical description comprise:
a location;
an imputation;
a feature flag; and
a whitelist flag.
10 . The method of claim 1 , wherein the schema is generated from:
user input; and
metadata associated with the features.
11 . The method of claim 1 , wherein the set of features comprises:
a member feature for a member of a social network;
a company feature for a company; and
a job feature for a job at the company.
12 . A system, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
obtain a set of features for use by a set of statistical models;
generate a schema comprising:
a logical description of data represented by the features; and
a physical description related to generating and storing the features;
output the schema for use in managing and sharing the features across the statistical models; and
update the outputted schema to reflect one or more parameters from a user.
13 . The system of claim 12 , wherein obtaining the set of features comprises:
obtaining a portion of the schema from one or more users; and
using the portion of the schema to automatically generate the set of features.
14 . The system of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
monitor one or more attributes associated with generating and using the features; and
output the one or more attributes with the schema.
15 . The system of claim 14 , wherein the one or more attributes comprise at least one of:
a recency;
a usage; and
a distribution.
16 . The system of claim 12 , wherein the logical description and the physical description of the schema comprise:
feature-level attributes that describe a feature in the set of features; and
feature-set-level attributes that describe the set of features.
17 . The system of claim 16 , wherein the feature-set-level attributes comprise at least one of:
a name;
a category;
a description;
one or more entities;
one or more tags;
one or more owners;
a location;
a format;
a frequency of generation;
a retention period;
a data availability delay;
a status; and
a source.
18 . The system of claim 16 , wherein the feature-level attributes comprise at least one of:
a location;
an imputation;
a feature flag;
a whitelist flag;
a name;
a namespace;
a description;
a feature type;
a data type;
an aggregation attribute; and
a transformation option.
19 . A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:
obtaining a set of features for use by a set of statistical models;
generating a schema comprising:
a logical description of data represented by the features; and
a physical description related to generating and storing the features;
outputting the schema for use in managing and sharing the features across the statistical models; and
updating the outputted schema to reflect one or more parameters from a user.
20 . The system of claim 19 , wherein obtaining the set of features comprises:
obtaining a portion of the schema from one or more users; and
using the portion of the schema to generate the set of features.