Processing real-time processing requests using machine learning models
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for providing consistent processing in a machine learning system are disclosed. A real-time processing request may be received and processed by both a preferred machine learning model and a fallback machine learning model. Processing for the preferred machine learning model may include obtaining additional information. A determination may be made regarding whether the processing of the real-time request by the preferred machine learning model has completed as of an expiration of an acceptable latency period. If the preferred model has not completed as of the expiration of an acceptable latency period, the response to the real-time request may be generated from the fallback model output. If the preferred model has completed prior to or by the expiration of the acceptable latency period, the response to the request may be generated from the preferred model output.
1. A computer-implemented method comprising:
receiving a first real-time processing request for first response data to be provided within an acceptable latency period beginning when the first real-time processing request is received at a machine learning system, the machine learning system comprising local memory that is local to the machine learning system, a fallback machine learning model, and a preferred machine learning model, wherein the fallback machine learning model and the preferred machine learning model are trained to perform a same particular machine learning task;
processing, by the machine learning system, the first real-time processing request by using the preferred machine learning model to generate a preferred model output and by using the fallback machine learning model in parallel to generate a fallback model output;
wherein processing by using the preferred machine learning model comprises:
executing a remote call to obtain additional information stored at least in part on memory remotely located from the machine learning system, and
processing, as input to the preferred machine learning model, the first real-time processing request and the additional information obtained by the remote call, to generate the preferred model output;
wherein processing by using the fallback machine learning model comprises:
processing, as input to the fallback machine learning model, the first real-time processing request and approximated information, to generate the fallback model output within the acceptable latency period, wherein the approximated information is stored in the local memory and comprises an approximation of the additional information, and wherein the fallback machine learning model processes the first real-time processing request using only request data in the first real-time processing request or data previously known to and locally stored in the machine learning system;
determining, by the machine learning system, within the acceptable latency period, whether the processing of the first real-time processing request by the preferred machine learning model has completed to generate the preferred model output, and, if so,
generating the first response data from the preferred model output, and if not, generating the first response data from the fallback model output; and
providing the first response data as a response to the first real-time processing request.
2. The computer-implemented method of claim 1 , wherein the acceptable latency period is configurable.
3. The computer-implemented method of claim 1 , wherein the acceptable latency period is predetermined prior to processing the first real-time processing request.
4. The computer-implemented method of claim 1 ,
wherein the first real-time processing request is a request to predict a next transaction price or a next transaction price direction for one or more financial products, and
wherein the additional information comprises information about transaction prices in a last few milliseconds.
5. The computer-implemented method of claim 4 , wherein the financial products comprise one or more of common stock shares, exchange traded fund shares, options contracts, commodity futures contracts, or financial derivatives.
6. The computer-implemented method of claim 1 ,
wherein the first real-time processing request is a request to predict whether a particular transaction is anomalous or fraudulent, and
wherein the additional information comprises information about transactions.
7. The computer-implemented method of claim 6 , wherein:
the particular transaction is a completed credit card or debit card transaction, and the additional information includes information regarding credit card or debit card transactions completed in a last few minutes.
8. The computer-implemented method of claim 6 , wherein:
the particular transaction is a financial claims processing transaction, and
the additional information includes information regarding financial claims processing transactions.
9. The computer-implemented method of claim 6 , wherein:
the particular transaction is an insurance claims transaction, a healthcare claims transaction, or an employee expense claims transaction, and
the additional information includes information regarding claims transactions.
10. The computer-implemented method of claim 1 ,
wherein the first real-time processing request is a request to predict security threats or anomalous behavior for one or more users or machines on a network, and
wherein the additional information comprises information about user or machine behavior.
11. A system comprising:
one or more computers; and
one or more memory devices including local memory that is local to the one or more computers, wherein the one or more memory devices store instructions that are operable, when executed by the one or more computers, to cause the one or more computers, performing as a machine learning system, to perform operations comprising:
receiving a first real-time processing request for first response data to be provided within an acceptable latency period beginning when the first real-time processing request is received at the machine learning system, the machine learning system comprising the local memory, a fallback machine learning model, and a preferred machine learning model, wherein the fallback machine learning model and the preferred machine learning model are trained to perform a same particular machine learning task;
processing, by the machine learning system, the first real-time processing request by using the preferred machine learning model to generate a preferred model output and by using the fallback machine learning model in parallel to generate a fallback model output;
wherein processing by using the preferred machine learning model comprises;
executing a remote call to obtain additional information stored at least in part on memory remotely located from the machine learning system, and
processing, as input to the preferred machine learning model, the first real-time processing request and the additional information obtained by the remote call to generate the preferred model output;
wherein processing by using the fallback machine learning model comprises:
processing, as input to the fallback machine learning model, the first real-time processing request and approximated information to generate flail the fallback model output within the acceptable latency period, wherein the approximated information is stored in the local memory and comprises an approximation of the additional information, and wherein the fallback machine learning model processes the first real-time processing request using only request data in the first real-time processing request or data previously known to and locally stored in the machine learning system;
determining, by the machine learning system, within the acceptable latency period, whether the processing of the first real-time processing request by the preferred machine learning model has completed to generate the preferred model output and, if so, generating the first response data from the preferred model output, and if not, generating the first response data from the fallback model output; and
providing the first response data as a response to the first real-time processing request.
12. The system of claim 11 , wherein the acceptable latency period is configurable.
13. The system of claim 11 , wherein the acceptable latency period is predetermined prior to processing the first real-time processing request.
14. One or more non-transitory computer-readable storage mediums comprising instructions stored thereon that are executable by a processing device and upon such execution cause the processing device to perform machine learning system operations comprising:
receiving a first real-time processing request for first response data to be provided within an acceptable latency period beginning when the first real-time processing request is received at a machine learning system, the machine learning system comprising local memory that is local to the machine learning system, a fallback machine learning model and a preferred machine learning model wherein the fallback machine learning model and the preferred machine learning model are trained to perform a same particular machine learning task;
processing, by the machine learning system, the first real-time processing request by using the preferred machine learning model to generate a preferred model output and by using the fallback machine learning model in parallel to generate a fallback model output;
wherein processing by using the preferred machine learning model comprises:
executing a remote call to obtain additional information stored at least in part on memory remotely located from the machine learning system, and
processing, as input to the preferred machine learning model, the first real-time processing request and the additional information obtained by the remote call to generate the preferred model output;
wherein processing by using the fallback machine learning model comprises:
processing, as input to the fallback machine learning model, the first real-time processing request and approximated information to generate the fallback model output within the acceptable latency period, wherein the approximated information is stored in the local memory and comprises an approximation of the additional information, and wherein the fallback machine learning model processes the first real-time processing request using only request data in the first real-time processing request or data previously known to and locally stored in the machine learning system;
determining, by the machine learning system, within the acceptable latency period, whether the processing of the first real-time processing request by the preferred machine learning model has completed to generate the preferred model output, and, if so,
generating the first response data from the preferred model output, and if not, generating the first response data from the fallback model output;
and
providing the first response data as a response to the first real-time processing request.