IP Library › Granted Patent US 10,878,144
Granted Patent B2
US 10,878,144 · App. 15/673,872 · Granted Dec 29, 2020

Multi-platform model processing and execution management engine

Inventors: Robert Andrew Nendorf (Chicago, IL); Nilesh Malpekar (Lincolnshire, IL); Mark V. Slusar (Chicago, IL); Joseph Alan Kleinhenz (Bolingbrook, IL); Robert Andrew Kreek (Seattle, WA); Patrick O'Reilly (Belfast, GB)
Assignee: Allstate Insurance Company
G06F30/20G06F9/46
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,878,144
App. No.
15/673,872
Granted
Dec 29, 2020
Kind
B2
Abstract

Systems and methods are disclosed for managing the processing and execution of models that may have been developed on a variety of platforms. A multi-model execution module specifying a sequence of models to be executed may be determined. A multi-platform model processing and execution management engine may execute the multi-model execution module internally, or outsource the execution to a distributed model execution orchestration engine. A model data monitoring and analysis engine may monitor the internal and/or distributed execution of the multi-model execution module, and may further transmit notifications to various computing systems.

Claims (92)

1. An apparatus comprising:

a processor; and

memory storing computer-executable instructions that, when executed by the processor, cause the apparatus to:

receive a first external machine learning model generated in a first modeling framework;

receive a second external machine learning model generated in a second modeling framework distinct from the first modeling framework;

generate a first machine learning model by processing the first external machine learning model, wherein the first machine learning model is formatted using a common modeling framework;

generate a second machine learning model by processing the second external machine learning model, wherein the second machine learning model is formatted using the common modeling framework;

transmit the first machine learning model to a mobile device;

trigger execution of the first machine learning model by the mobile device;

receive a dataset generated based on execution of the first machine learning model by the mobile device;

obtain a historical statistical distribution generated based on the dataset;

calculate a statistical distribution based on the dataset;

determine a shift in the distribution of values between the historical statistical distribution and the statistical distribution;

transmit a notification indicating the shift in distribution of values based on the shift exceeding a threshold value;

automatically determine, based on the dataset and the shift in distribution of values exceeding the threshold value, a second machine learning model for execution;

transmit, to a cloud computing device, the dataset;

trigger execution of the second machine learning model by the cloud computing device, wherein the execution of the second machine learning model is based on the dataset;

obtain a second dataset from the cloud computing device, wherein the second dataset is generated based on the execution of the second machine learning model; and

generate an aggregated dataset based on the dataset and the second dataset.

2. The apparatus of claim 1 , the memory storing computer-executable instructions that, when executed by processor, further cause the apparatus to:

identify a location of a first dataset to be used as input data by the first machine learning model;

retrieve the first dataset from the location; and

transmit the first dataset to the mobile device, wherein the dataset obtained from the mobile device is generated based on the first dataset.

3. The apparatus of claim 1 , wherein the first modeling framework is the same as the common modeling framework.

4. The apparatus of claim 1 , the memory storing computer-executable instructions that, when executed by processor, further cause the apparatus to:

determine that a portion of the dataset is to be used as input by one or more additional machine learning models; and

responsive to the determination that the portion of the dataset is to be used as input by the one or more additional machine learning models, update at least one dataset associated with the one or more additional machine learning models with the portion of the dataset.

5. The apparatus of claim 1 , wherein the instructions, when executed by processor, further cause the apparatus to transmit the aggregated dataset to the mobile device.

6. The apparatus of claim 1 , wherein the instructions, when executed by processor, further cause the apparatus to store the dataset in one or more external storage devices.

7. A method comprising:

obtaining, by a computing device, a first external machine learning model generated in a first modeling framework;

obtaining, by the computing device, a second external machine learning model generated in a second modeling framework distinct from the first modeling framework;

generating, by the computing device, a first machine learning model by processing the first external machine learning model, wherein the first machine learning model is formatted using a common modeling framework;

generating, by the computing device, a second machine learning model by processing the second external machine learning model, wherein the second machine learning model is formatted using the common modeling framework;

transmitting, by the computing device and to a mobile device, the first model;

triggering, by the computing device, execution of the first machine learning model by the mobile device;

receiving, by the computing device and from the mobile device, a dataset generated based on execution of the first machine learning model by the mobile device;

obtaining, by the computing device, a historical statistical distribution generated based on the dataset;

calculating, by the computing device, a statistical distribution based on the dataset;

determining, by the computing device, a shift in the distribution of values between the historical statistical distribution and the statistical distribution;

transmitting, by the computing device, a notification indicating the shift in distribution of values based on the shift exceeding a threshold value;

automatically determining, by the computing device and based on the dataset and the shift in distribution of values exceeding the threshold value, a second machine learning model for execution;

transmitting, by the computing device, the dataset to a cloud computing device;

triggering, by the computing device, execution of the second machine learning model by the cloud computing device, wherein the execution of the second machine learning model is based on the dataset;

obtaining, by the computing device, a second dataset from the cloud computing device, wherein the second dataset is generated based on the execution of the second machine learning model; and

generating, by the computing device, an aggregated dataset based on the dataset and the second dataset.

8. The method of claim 7 , further comprising:

identifying, by the computing device, a location of a first dataset to be used as input data by the first machine learning model;

retrieving, by the computing device, the first dataset from the location; and

transmitting, by the computing device, the first dataset to the mobile device, wherein the dataset obtained from the mobile device is generated based on the first dataset.

9. The method of claim 7 , further comprising:

receiving, by the computing device, a first external machine learning model generated in a first modeling framework;

receiving, by the computing device, a second external machine learning model generated in a second modeling framework distinct from the first modeling framework;

generating, by the computing device, the first machine learning model by processing the first external machine learning model, wherein the first machine learning model is formatted using the first modeling framework; and

generating, by the computing device, the second machine learning model by processing the second external machine learning model, wherein the second machine learning model is formatted using the first modeling framework.

10. The method of claim 7 , further comprising:

determining, by the computing device, that a portion of the dataset is to be used as input by one or more additional machine learning models; and

based on the determining, updating, by the computing device, at least one dataset associated with the one or more additional machine learning model with the portion of the dataset.

11. The method of claim 7 , further comprising transmitting, by the computing device, the aggregated dataset to the mobile device.

12. The method of claim 7 , further comprising storing, by the computing device, the dataset in one or more external storage devices.

13. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:

obtaining a first external machine learning model generated in a first modeling framework;

obtaining a second external machine learning model generated in a second modeling framework distinct from the first modeling framework;

generating a first machine learning model by processing the first external machine learning model, wherein the first machine learning model is formatted using a common modeling framework;

generating a second machine learning model by processing the second external machine learning model, wherein the second machine learning model is formatted using the common modeling framework;

transmitting the first machine learning model to a mobile device;

triggering execution of the first machine learning model by the mobile device;

receiving a dataset from the mobile device, wherein the dataset is based on execution of the first machine learning model by the mobile device;

obtaining a historical statistical distribution generated based on the dataset;

calculating a statistical distribution based on the dataset;

determining a shift in the distribution of values between the historical statistical distribution and the statistical distribution;

transmitting a notification indicating the shift in distribution of values based on the shift exceeding a threshold value;

automatically determining, based on the dataset and the shift in distribution of values exceeding the threshold value, a second machine learning model for execution;

transmitting the dataset to a cloud computing device;

triggering execution of the second machine learning model by the cloud computing device, wherein the execution of the second machine learning model is based on the dataset;

obtaining a second dataset from the cloud computing device, wherein the second dataset is generated based on the execution of the second machine learning model; and

generating an aggregated dataset based on the dataset and the second dataset.

14. The non-transitory computer readable medium of claim 13 , further storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:

identifying a location of a first dataset to be used as input data by the first machine learning model;

retrieving the first dataset from the location; and

transmitting the first dataset to the mobile device, wherein the results obtained from the mobile device are generated based on the first dataset.

15. The non-transitory computer readable medium of claim 13 , wherein the first modeling framework is the same as the common modeling framework.

16. The non-transitory computer readable medium of claim 13 , further storing instructions that, when executed by the one or more processors, cause the one or more processors to perform steps comprising:

determining that a portion of the dataset is to be used as input by one or more additional machine learning models; and

based on the determining, updating at least one dataset associated with the one or more additional machine learning model with the portion of the dataset.

17. The non-transitory computer readable medium of claim 13 , wherein the instructions that, when executed by the one or more processors, cause the one or more processors to perform steps comprising transmitting the aggregated dataset to the mobile device.

18. The apparatus of claim 1 , wherein the instructions, when executed by the processor, further cause the apparatus to:

automatically determine, based on the second dataset, a third machine learning model for execution;

transmit, to the cloud computing device, the second dataset;

trigger execution of the third machine learning model by the cloud computing device, wherein the execution of the third machine learning model is based on the second dataset;

obtain a third dataset from the cloud computing device, wherein the third dataset is generated based on the execution of the third machine learning model; and

generate the aggregated dataset further based on the third dataset.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 30, 2017
From: NENDORF, ROBERT ANDREW; MALPEKAR, NILESH; SLUSAR, MARK V.; KLEINHENZ, JOSEPH ALAN; KREEK, ROBERT ANDREW; O'REILLY, PATRICK
To: ALLSTATE INSURANCE COMPANY
Reel/Frame 044256/0477 →
Continuity (1)
Related Publication 20190050505A1 · Feb 14, 2019
Cited By (2)
US 12,586,004 US 12,645,693