Systems and methods for machine learning dataset generation
In some aspects, methods and systems may use consumption timestamps to ensure consistency between data used during inference and data used to train a machine learning model. Specifically, methods and systems may generate a consumption timestamp corresponding to the time at which the streaming data is received or inserted into a database. The consumption timestamp may be used as a synchronization point for any related data that may be requested (e.g., via an API). After streaming data is received, the consumption timestamp may be generated and assigned to any data that is requested in connection with the streaming data. The streaming data, related data, and consumption timestamp may be included in a data snapshot that is provided to a machine learning model for inference (e.g., real-time inference) or training.
1 . A machine learning data service system for synchronizing datasets by a consumption timestamp to provide consistent inference by machine learning models, the system comprising:
one or more processors programmed with computer program instructions that, when executed by the one or more processors, cause operations comprising:
in response to receiving, via a real-time streaming service, a streaming financial transaction data comprising (i) an indication of one or more actions performed by a user and (ii) a first timestamp indicating when the streaming financial transaction data was generated, sending, via an application programming interface (API), a request for API data associated with the user;
in response to receiving the API data associated with the user, generating a data snapshot comprising modified streaming data and the API data that are synchronized with a consumption timestamp indicating a time of the receiving of the streaming financial transaction data in lieu of the first timestamp indicating when the streaming financial transaction data was generated such that the modified streaming data includes the streaming financial transaction data without the first timestamp; and
inputting the data snapshot to a machine learning model to obtain a prediction corresponding to the data snapshot.
2 . A method comprising:
receiving, via a streaming service, a streaming data comprising an indication of one or more events that have occurred;
sending, via an application programming interface (API) and based on the streaming data, a request for API data associated with the one or more events;
generating a data snapshot comprising the streaming data, the API data, and a consumption timestamp (i) indicating a time of the receiving of the streaming data and (ii) associated with the streaming data and the API data; and
based on providing the data snapshot to a machine learning model, generating a prediction corresponding to the data snapshot.
3 . The method of claim 2 , wherein generating the data snapshot comprises associating the consumption timestamp, indicating the time of the receiving of the streaming data, with the streaming data in lieu of a creation timestamp corresponding to a time at which the streaming data was generated.
4 . The method of claim 2 , wherein generating the data snapshot comprises generating the data snapshot to comprise the streaming data, the API data, and the consumption timestamp that corresponds to an expiration time of a threshold amount of time after receiving a first portion of the streaming data, despite receiving the first portion before the expiration time.
5 . The method of claim 2 , wherein sending the request for the API data comprises sending, based on the streaming data and the consumption timestamp being the time of the receiving of the streaming data, an API query comprising (i) an identifier associated with the one or more events and (ii) the consumption timestamp in lieu of an earlier timestamp corresponding to a time at which the streaming data was generated.
6 . The method of claim 2 , wherein the streaming data is received asynchronously from a plurality of computing devices associated with the API.
7 . The method of claim 2 , further comprising:
after generating the prediction corresponding to the data snapshot, retraining the machine learning model using the data snapshot;
generating a second data snapshot based on second streaming data and a second consumption timestamp; and
generating, via the retrained machine learning model, a given prediction corresponding to the second data snapshot.
8 . The method of claim 2 , further comprising:
causing, based on the prediction corresponding to the data snapshot, a modification to a user interface to increase a likelihood of a user performing a target action.
9 . The method of claim 2 , wherein the one or more events comprise one or more actions performed by a user, and wherein the API data comprises demographic information associated with the user.
10 . The method of claim 2 , wherein generating the prediction corresponding to the data snapshot comprises:
inputting a portion of the data snapshot into the machine learning model; and
generating, via the machine learning model, output that predicts a user's next action.
11 . The method of claim 2 , further comprising:
based on the data snapshot, displaying, on a user interface, the consumption timestamp and a portion of the streaming data.
12 . One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause operations comprising:
receiving, via a streaming service, a streaming data comprising an indication of one or more events that have occurred;
based on receiving the streaming data, generating a consumption timestamp indicating a time at which the streaming data is received;
sending, via an application programming interface (API) and based on receiving the streaming data, a request for API data associated with the one or more events;
based on the request for API data, receiving the API data associated with the one or more events;
generating a data snapshot comprising the streaming data, the consumption timestamp, and the API data; and
based on providing the data snapshot to a machine learning model, generating a prediction corresponding to the data snapshot.
13 . The one or more non-transitory computer-readable media of claim 12 , wherein generating a data snapshot comprises associating the consumption timestamp, indicating the time of the receiving of the streaming data, with the streaming data in lieu of a creation timestamp corresponding to a time at which the streaming data was generated.
14 . The one or more non-transitory computer-readable media of claim 12 , wherein generating the data snapshot comprises generating the data snapshot to comprise the streaming data, the API data, and the consumption timestamp that corresponds to an expiration time of a threshold amount of time after receiving a first portion of the streaming data, despite receiving the first portion before the expiration time.
15 . The one or more non-transitory computer-readable media of claim 12 , wherein sending the request for API data comprises sending, based on the streaming data and the consumption timestamp being the time of the receiving of the streaming data, an API query comprising (i) an identifier associated with the one or more events and (ii) the consumption timestamp in lieu of an earlier timestamp corresponding to a time at which the streaming data was generated.
16 . The one or more non-transitory computer-readable media of claim 12 , wherein the streaming data is received asynchronously from a plurality of computing devices associated with the API.
17 . The one or more non-transitory computer-readable media of claim 12 , further comprising:
after generating the prediction corresponding to the data snapshot, retraining the machine learning model using the data snapshot;
generating a second data snapshot based on second streaming data and a second consumption timestamp; and
generating, via the retrained machine learning model, a given prediction corresponding to the second data snapshot.
18 . The one or more non-transitory computer-readable media of claim 12 , the operations further comprising:
causing, based on the prediction corresponding to the data snapshot, a modification to a user interface to increase a likelihood of a user performing a target action.
19 . The one or more non-transitory computer-readable media of claim 12 , wherein the one or more events comprise one or more actions performed by a user, and wherein the API data comprises demographic information associated with the user.
20 . The one or more non-transitory computer-readable media of claim 12 , wherein generating the prediction corresponding to the data snapshot comprises:
inputting a portion of the data snapshot into the machine learning model; and
generating, via the machine learning model, output that predicts a user's next action.