Real time fault tolerant stateful featurization
Certain aspects of the present disclosure provide techniques for operation of a feature management platform. A feature management platform is an end-to-end platform developed to manage the full lifecycle of data features. For example, to create a stateful feature, the feature management platform can receive a processing artifact from a computing device. The processing artifact defines the stateful feature, including the data source to retrieve event data from, when to retrieve the event data, the type of transform to apply, etc. Based on the processing artifact, the feature management system generates a processing job (e.g., the API defines a pipeline), which when initiated generates a vector that encapsulates the stateful feature. The vector is transmitted to the computing device that locally hosts a model, which generates a prediction that is transmitted to the feature management platform. Subsequently, the predication and stateful feature can be transmitted to other computing devices.
1. A method, comprising:
receiving, from a computing device, a processing artifact defining a stateful feature including:
a data source to retrieve event data; and
a transform to apply to the event data from the data source;
generating, based on the processing artifact, a processing job;
initiating the processing job, wherein the processing job includes:
retrieving the event data from the data source;
applying the transform to the event data to generate a set of feature values;
aggregating the set of feature values in an external cache;
upon aggregating the set of feature values in the external cache:
retrieving the aggregated set of feature values in the external cache;
generating a stateful feature based on the aggregated set of feature values;
encapsulating the stateful feature within a vector;
providing the vector to the computing device that hosts a model;
registering the stateful feature and associated metadata in a feature registry;
receiving, from the computing device, a prediction generated by the model hosted on the computing device;
transmitting the prediction to a second computing device;
receiving, from an additional computing device, a request for the stateful feature;
determining, based on locating the stateful feature in the feature registry using the associated metadata, not to re-generate the stateful feature in response to the request; and
providing, to the additional computing device, the stateful feature in response to the request.
2. The method of claim 1 , wherein the method further comprises: backfilling the external cache with the set of feature values to generate the stateful feature.
3. The method of claim 1 , further comprising: providing a user interface via an API to the computing device to define the stateful feature.
4. The method of claim 1 , further comprising: storing the set of feature values in the external cache.
5. The method of claim 1 , wherein aggregating the set of feature values is based on input data from the computing device.
6. The method of claim 1 , further comprising receiving a configuration file from the computing device.
7. The method of claim 6 , wherein the configuration file includes input data defining the stateful feature.
8. The method of claim 7 , wherein the input data includes an aggregation operation for generating the stateful feature.
9. A system, comprising:
a processor; and
a memory storing instructions, which when executed by the processor perform a method comprising:
receiving, from a computing device, a processing artifact defining a stateful feature including:
a data source to retrieve event data; and
a transform to apply to the event data from the data source;
generating, based on the processing artifact, a processing job;
initiating the processing job, wherein the processing job includes:
retrieving the event data from the data source;
applying the transform to the event data to generate a set of feature values;
aggregating the set of feature values in an external cache;
upon aggregating the set of feature values in the external cache:
retrieving the aggregated set of feature values in the external cache;
generating a stateful feature based on the aggregated set of feature values;
encapsulating the stateful feature within a vector; and
providing the vector to the computing device that hosts a model;
registering the stateful feature and associated metadata in a feature registry;
receiving, from the computing device, a prediction generated by the model hosted on the computing device;
transmitting the prediction to a second computing device;
receiving, from an additional computing device, a request for the stateful feature;
determining, based on locating the stateful feature in the feature registry using the associated metadata, not to re-generate the stateful feature in response to the request; and
providing, to the additional computing device, the stateful feature in response to the request.
10. The system of claim 9 , wherein the method further comprises: backfilling the external cache with the set of feature values to generate the stateful feature.
11. The system of claim 9 , wherein the method further comprises: providing a user interface via an API to the computing device to define the stateful feature.
12. The system of claim 9 , wherein the method further comprises: storing the set of feature values in the external cache.
13. The system of claim 9 , wherein aggregating the set of feature values is based on input data from the computing device.
14. The system of claim 9 , wherein the method further comprises receiving a configuration file from the computing device.
15. The system of claim 14 , wherein the configuration file includes input data defining the stateful feature.
16. The system of claim 15 , wherein the input data includes an aggregation operation for generating the stateful feature.
17. A method, comprising:
retrieving, at a feature management platform, a raw event from a streaming data source;
providing the raw event to a feature calculation logic of the feature management platform;
implementing the feature calculation logic on the raw event for a period of time matching a size of an aggregation time window to create a set of feature values;
storing the set of feature values in a cache of the feature management platform;
invoking an aggregation logic to generate a stateful feature;
registering the stateful feature and associated metadata in a feature registry;
receiving, from a computing device, a request for the stateful feature;
determining, based on locating the stateful feature in the feature registry using the associated metadata, not to re-generate the stateful feature in response to the request; and
providing, to the computing device, the stateful feature in response to the request.
18. The method of claim 17 , further comprising providing a set of user interfaces to collect input data defining a stateful feature.
19. The method of claim 18 , wherein the input data collected by a user interface in the set of user interfaces comprises:
the size of the aggregation time window; and
the feature calculation logic.
20. The method of claim 17 , further comprising: publishing the stateful feature in a feature queue.