Granted Patent B1

US 12,731,171 · App. 18/531,567 · Granted Sep 8, 2026

Inventors: Steven Ostrover (Los Angeles, CA); Jonathan Frederick Ripper (Los Angeles, CA)

Assignee: CPL Assets, LLC

G06Q30/0247G06F16/951G06N7/01

Abstract

Claims (74)

1 . A method comprising:

accessing inquiry data associated with a plurality of online inquiries from a plurality of user devices, the plurality of online inquiries comprising responses to electronic forms with predefined user-fillable fields prompting for a user to enter data, wherein the electronic forms are generated based on user responses to advertisements;

creating prepared inquiry data comprising eligible inquiries from the inquiry data by filtering out ineligible inquiries;

determining, based at least in part on the prepared inquiry data, a first model for predicting enrollment rates of inquiries by automatically testing a plurality of models by:

identifying historical enrollment data associated with a plurality of prior enrollment inquiries received within a first time period, wherein the historical enrollment data comprises actual enrollment results for each prior enrollment inquiry of the plurality of prior enrollment inquiries;

selecting a first subset of the historical enrollment data; and

running the first subset through each of the plurality of models to determine a predictive accuracy, wherein the predictive accuracy is determined for each model based on a difference between an output of each model and the actual enrollment results, the plurality of models including two or more of: a decision tree model, a random forest model, a regression model, a classification model, a regression tree model, or a neural network;

identifying the first model as having a highest predictive accuracy of the plurality of models based at least in part on the determined predictive accuracy for each of the plurality of models tested and one or more hardware processing capabilities of a special-purpose computing device configured to store and host the first model;

determining, using the historical enrollment data, predicted enrollment rates for the plurality of prior enrollment inquiries based at least in part on user data and the actual enrollment results associated with the prior enrollment inquiries;

updating the first model to create an updated first model based at least in part on the predicted enrollment rates, wherein the updated first model is configured to receive the inquiry data from an online inquiry of the plurality of online inquiries;

accessing a first inquiry of the plurality of online inquiries associated with a first user device of the plurality of user devices generated within a second time period, wherein the second time period begins after the first time period ends;

inputting, into the updated first model, first inquiry data corresponding to the first inquiry and comprising a media type and a device type, wherein the updated first model is a decision-tree architecture comprising a plurality of nodes and segments corresponding to the media type and device type;

applying the updated first model using classification rules to assign the first inquiry to an item, the item specifying a predicted enrollment rate value for a characteristic or group of characteristics corresponding to the media type or the device type, wherein the classification rules control how the updated first model processes the first inquiry data based on data availability and predictive reliability so that the predicted enrollment rate is assigned to the first inquiry based on the media type or device type associated with the first inquiry data, wherein the predicted enrollment rate indicates a likelihood of enrollment;

receiving, from the updated first model, output data comprising the predicted enrollment rate for the first inquiry;

in response to receiving an output from the updated first model, determining a potential revenue that can be generated from the first inquiry, wherein the potential revenue is based at least in part on the predicted enrollment rate and actual revenue generated from ineligible inquiries, wherein ineligible inquiries comprise incomplete inquiries; and

automatically adjusting, based on the potential revenue, an allocation of advertisements among a plurality of online delivery channels for presenting at least one or more products, services, or programs such that a new advertisement or a new online delivery channel is used based at least in part on results of the updated first model and on shared characteristics with prior inquiries, the new advertisement or new online delivery channel yielding an improved enrollment rate and potential revenue compared to advertisements or online delivery channels used prior to adjusting the allocation of advertisements.

2 . The method of claim 1 , wherein the first time period is based at least in part on how long it takes for the historical enrollment data to become complete by including actual enrollment data.

3 . The method of claim 1 , further comprising:

generating or updating the classification rules based at least in part on the updated first model.

4 . The method of claim 1 , wherein each of the plurality of online inquiries is associated with a quality level indicative of a likelihood of enrollment in a program.

5 . The method of claim 4 , wherein the quality level has an enrollment rate associated with the quality level.

6 . The method of claim 1 , wherein the updated first model further comprises a plurality of items, each of the plurality of items associated with one or more characteristics of the plurality of online inquiries, each of the plurality of items specifying respective predicted enrollment rate values for the associated one or more characteristics.

7 . The method of claim 6 , wherein each node specifies a predicted enrollment rate value for a group of characteristics associated with the plurality of online inquiries, wherein each segment specifies the predicted enrollment rate value for a particular characteristic associated with the plurality of online inquiries, and wherein the characteristics correspond to the media type or the device type.

8 . The method of claim 1 , further comprising:

access updated historical enrollment data indicating updates to the historical enrollment data of at least one of the plurality of online inquiries; and

determine, based at least in part on the updated historical enrollment data and the updated first model, a newly updated model for predicting enrollment rates of inquiries, wherein applying the updated first model to inquiry data associated with the first inquiry comprises applying the newly updated model to inquiry data associated with the first inquiry.

9 . The method of claim 1 , wherein the first time period includes one of: (1) a roll window or fixed period of time, or (2) an append window or increasing period of time.

10 . A system comprising:

one or more computer readable storage devices configured to store computer executable instructions; and

one or more hardware computer processors in communication with the one or more computer readable storage devices to execute the computer executable instructions to cause the system to:

access inquiry data associated with a plurality of online inquiries from a plurality of user devices, the plurality of online inquiries comprising responses to electronic forms with predefined user-fillable fields prompting for a user to enter data, wherein the electronic forms are generated based on user responses to advertisements;

create prepared inquiry data comprising eligible inquiries from the inquiry data by filtering out ineligible inquiries;

determine, based at least in part on the prepared inquiry data, a first model for predicting enrollment rates of inquiries by automatically testing a plurality of models by:

identifying historical enrollment data associated with a plurality of prior enrollment inquiries received within a first time period, wherein the historical enrollment data comprises actual enrollment results for each prior enrollment inquiry of the plurality of prior enrollment inquiries;

running data first subset of the historical enrollment data through each of the plurality of models to determine a predictive accuracy, wherein the predictive accuracy is determined for each model based on a difference between an output of each model and the actual enrollment results, the plurality of models including two or more of: a decision tree model, a random forest model, a regression model, a classification model, a regression tree model, or a neural network; and

identifying the first model as having a highest predictive accuracy of the plurality of models based at least in part on the determined predictive accuracy for each of the plurality of models tested and one or more hardware processing capabilities of a special-purpose computing device configured to store and host the first model;

determine, using the historical enrollment data, predicted enrollment rates for the plurality of prior enrollment inquiries;

update the first model to create an updated first model based at least in part on the predicted enrollment rates;

access a first inquiry of the plurality of online inquiries associated with a first user device of the plurality of user devices generated within a second time period, wherein the second time period begins after the first time period ends;

input, into the updated first model, first inquiry data corresponding to the first inquiry and comprising one or more characteristics, wherein the updated first model is a decision-tree architecture comprising a plurality of nodes and segments corresponding to the one or more characteristics;

apply the updated first model using classification rules to assign the first inquiry to an item, the item specifying a predicted enrollment rate value for a characteristic or group of characteristics corresponding to the one or more characteristics, wherein the classification rules control how the updated first model processes the first inquiry data based on data availability and predictive reliability so that the predicted enrollment rate is assigned to the first inquiry based on the one or more characteristics associated with the first inquiry data, wherein the predicted enrollment rate indicates a likelihood of enrollment;

receive, from the updated first model, output data comprising the predicted enrollment rate for the first inquiry;

in response to receiving an output from the updated first model, determine a potential revenue that can be generated from the first inquiry, wherein the potential revenue is based at least in part on the predicted enrollment rate and actual revenue generated from ineligible inquiries, wherein ineligible inquires comprise incomplete inquiries;

automatically adjust, based on the potential revenue, an allocation of advertisements among a plurality of online delivery channels for presenting at least one or more products, services, or programs such that a new advertisement or a new online delivery channel is used based at least in part on results of the updated first model and on shared characteristics with prior inquiries, the new advertisement or new online delivery channel yielding an improved enrollment rate and potential revenue compared to advertising or online delivery channels used prior to adjusting the allocation of advertisements.

11 . The system of claim 10 , further comprising:

generating or updating the classification rules based at least in part on the updated first model.

12 . The system of claim 10 , wherein each of the plurality of online inquiries is associated with a quality level indicative of a likelihood of enrollment in a program.

13 . The system of claim 10 , further comprising a plurality of items, each of the plurality of items associated with one or more characteristics of the plurality of online inquiries, each of the plurality of items specifying respective predicted enrollment rate values for the associated one or more characteristics.

14 . A method comprising:

accessing inquiry data associated with a plurality of online inquiries from a plurality of user devices, the plurality of online inquiries comprising responses to electronic forms with predefined user-fillable fields prompting for a user to enter data, wherein the electronic forms are generated based on user responses to advertisements;

creating prepared inquiry data comprising eligible inquiries from the inquiry data by filtering out ineligible inquiries;

identifying historical enrollment data associated with a plurality of prior enrollment inquiries received within a first time period, wherein the historical enrollment data comprising actual enrollment results for each prior enrollment inquiry of the plurality of prior enrollment inquiries;

determining a first model for predicting enrollment rates of inquiries, the first model having a highest predictive accuracy of a plurality of models based on applying a first subset of the historical enrollment data to each model of the plurality of models and one or more hardware processing capabilities of a special-purpose computing device configured to store and host the first model;

determining, using the historical enrollment data, predicted enrollment rates for the plurality of prior enrollment inquiries;

updating the first model to create an updated first model based at least in part on the predicted enrollment rates;

accessing a first inquiry of the plurality of online inquiries associated with a first user device of the plurality of user devices generated within a second time period, wherein the second time period begins after the first time period ends;

inputting, into the updated first model, first inquiry data corresponding to the first inquiry and comprising one or more characteristics, wherein the updated first model is a decision-tree architecture comprising a plurality of nodes and segments corresponding to the characteristics;

applying the updated first model using classification rules to assign the first inquiry to an item, the item specifying a predicted enrollment rate value for a characteristic or group of characteristics corresponding to the one or more characteristics, wherein the classification rules control how the updated first model processes the first inquiry data based on data availability and predictive reliability so that the predicted enrollment rate is assigned to the first inquiry based on the one or more characteristics associated with the first inquiry data, wherein the predicted enrollment rate indicates a likelihood of enrollment;

receiving, from the updated first model, output data comprising the predicted enrollment rate for the first inquiry;

in response to receiving an output from the updated first model, determining a potential revenue that can be generated from the first inquiry, wherein the potential revenue is based at least in part on the predicted enrollment rate and actual revenue generated from ineligible inquiries, wherein ineligible inquiries comprise incomplete inquiries; and

automatically adjusting, based on the potential revenue, an allocation of advertisements among a plurality of online delivery channels for presenting at least one or more products, services, or programs such that a new advertisement or a new online delivery channel is used based at least in part on results of the updated first model and on shared characteristics with prior inquiries, the new advertisement or new online delivery channel yielding an improved enrollment rate and potential revenue compared to advertisements or online delivery channels used prior to adjusting the allocation of advertisements.

15 . The method of claim 14 , wherein data associated with one or more bots is removed from the inquiry data by one or more of: comparing a speed at which an inquiry is filled out to a threshold value, including links and fields that are not visible to the human eye, or evaluating a distribution of clicks across an advertisement.

16 . The method of claim 14 , wherein applying the updated first model to the inquiry data is based at least in part on classification rules, wherein the classification rules indicate how the first inquiry should be categorized in the updated first model so that the predicted enrollment rate is assigned to the inquiry data associated with the first inquiry, wherein the predicted enrollment rate indicates a likelihood of enrollment.

17 . The method of claim 1 , wherein the adjusting one or more parameters for the first model comprises:

(i) for a decision tree model, adjusting one or more of: a splitting criterion, a minimum number of observations for a leaf node, a maximum number of branches from a node, or a maximum depth of the tree;

(ii) for a random forest model, adjusting one or more of: a number of trees in an ensemble, a number of features considered when splitting a node, a minimum number of samples required to split an internal node, or a maximum tree depth for each tree in the ensemble;

(iii) for a regression model, adjusting one or more of: a regularization parameter, a learning rate, or a feature selection threshold;

(iv) for a classification model, adjusting one or more of: a class weighting scheme, a threshold for class assignment, or a regularization term;

(v) for a regression tree model, adjusting one or more of: a minimum number of observations per split, a pruning strategy, or a maximum depth of the tree; and

(vi) for a neural network model, adjusting one or more of: a number of hidden layers, a number of neurons per layer, an activation function, a learning rate, a batch size, or a number of training epochs.

18 . The method of claim 1 , wherein the special-purpose computing device includes an application-specific integrated circuit or a field programmable gate array.

19 . The method of claim 1 , wherein the media type includes one or more of: social media, a web-page display, or a search engine.

20 . The method of claim 14 , further comprising:

generating or updating the classification rules based at least in part on the updated first model.

Assignments (2)

Continuity (5)

References Cited (103)