Granted Patent B2

US 11,354,700 · App. 16/453,471 · Granted Jun 7, 2022

Inventors: Pavan Korada (San Mateo, CA); Sunpreet Singh Khanuja (Santa Clara, CA); Weiwei Zhang (Palo Alto, CA); Bharat Goyal (San Jose, CA)

Assignee: Zeta Global Corp.

G06Q30/0243G06F16/24578G06F16/9535G06F30/20G06Q30/016G06Q30/0251

Abstract

Claims (75)

1. A system comprising at least one processor and a memory in communication with the at least one processor, wherein the system is programmed to:

receive from a lead generator component first execution cycle data comprising a first set of leads and a second set of leads, wherein the first set of leads comprises first lead information describing a plurality of first potential customers and the second set of leads comprises second lead information describing a plurality of second potential customers;

determine by a lead scorer component a first lead score for the plurality of first potential customers in the first set of leads based at least in part on a first lead scoring model;

determine that the first lead score for the first lead is greater than a first lead score threshold for the plurality of first potential customers in the first set of leads;

determine by the lead scorer component that a second lead score for the plurality of second potential customers in the second set of leads is less than the first lead score threshold;

filter a set of filtered leads including the first lead information from the plurality of first potential customers in the first set of leads;

determine by an evaluator component that a quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold, the analysis window data comprising a plurality of leads received during an analysis window time period;

determine a second lead scoring model based at least in part on the first execution cycle data;

receive second execution cycle data comprising a third set of leads comprising third lead information describing a plurality of third potential customers;

determine by the lead scorer component a third lead score for the plurality of third potential customers in the third set of leads based at least in part on the second lead scoring model wherein the lead scorer component determines whether a guard-rail enforcer component controls the determined second lead scoring model;

determine that the third lead score for the plurality of third potential customers in the set of third leads is greater than the first lead score threshold;

filter a second set of filtered leads including the third lead information from the third lead; and

adjust at least one of the scoring models on the lead scorer component based on the quality of leads determined by the plurality of lead scoring models;

wherein the determining that the quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold comprises:

determining a scrub rate, wherein the scrub rate describes a portion of the first execution cycle data having lead scores greater than the first lead score threshold;

determine that the scrub rate is greater than an analysis window scrub rate by more than a scrub rate threshold; and

wherein the system is further programmed to receive lead score data describing lead scores for at least one lead received during an observation period and at least one lead received during the analysis window and not during the observation period.

2. The system of claim 1 , wherein the determining the second lead scoring model comprises re-training the first lead score model based at least in part on the first execution cycle data.

3. The system of claim 1 , wherein the determining the second lead scoring model comprises modifying a scoring parameter of the first lead score model.

4. The system of claim 1 , further programmed to after determining the second lead scoring model, determine that application of the second lead scoring model is permitted by a limitation rule.

5. The system of claim 1 , further programmed to:

determine an average scrub rate for a second plurality of leads received during the analysis window; and

determine the scrub rate threshold based at least in part on the average scrub rate.

6. The system of claim 1 , further programmed to:

determine a mean scrub rate of a plurality of historical scrub rates to generate the analysis window scrub rate; and

determine that the scrub rate threshold is more than two standard deviations higher than the mean scrub rate.

7. The system of claim 1 , wherein determining that the quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold comprises determining that a first value of the first lead information indicates that the first lead is not likely to convert.

8. The system of claim 1 , where the system is further programmed to: before generating the second set of filtered leads, send a message to an administrative user; and

determine that more than a threshold time has passed since the sending of the message.

9. The system of claim 1 , wherein the first execution cycle data consists of leads received during an observation period shorter than the analysis window.

10. The system of claim 1 , wherein the system is further programmed to determine a scrub rate for a plurality of lead scores for leads received during the analysis window.

11. The system of claim 1 , wherein the first execution cycle data and the second execution cycle data consist of leads with a first common value for a first lead category, and wherein the first execution cycle data and the second execution cycle data consist of leads with a second common value for a second common lead value.

12. A method for adaptively generating leads, comprising:

receiving first execution cycle data comprising a first set of leads and a second set of leads, wherein the first set of leads comprises first lead information describing a first plurality of potential customers and the second set of leads comprises second lead information describing a second plurality of potential customers;

determining a first lead score for each of the first plurality of potential customers in the first set of leads based at least in part on a first lead scoring model;

determining that the first lead scores for the first set of leads is greater than a first lead score threshold;

determining that a second lead score for the second set of leads is less than the first lead score threshold;

filtering a set of filtered leads including the first lead information from the first set of leads;

determining that a quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold, the analysis window data comprising a plurality of leads received during an analysis window time period;

determining a second lead scoring model based at least in part on the first execution cycle data;

receiving second execution cycle data comprising a third set of leads comprising third lead information describing a plurality of third potential customers;

determining a third lead score for each of the plurality of third potential customers in the third set of leads based at least in part on the second lead scoring model including determining whether a guard-rail enforcer component controls the determined second lead scoring model;

determining that the third lead score for the third lead is greater than the first lead score threshold;

selecting a second set of filtered leads including the third lead information from the plurality of third leads; and

adjust at least one of the scoring models on the lead scorer component based on the quality of leads determined by the plurality of lead scoring models;

wherein the determining that the quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold comprises:

determining a scrub rate, wherein the scrub rate describes a portion of the first execution cycle data having lead scores greater than the first lead score threshold;

determine that the scrub rate is greater than an analysis window scrub rate by more than a scrub rate threshold; and

receiving lead score data describing lead scores for at least one lead received during an observation period and at least one lead received during the analysis window and not during the observation period.

13. The method of claim 12 , wherein the determining the second lead scoring model comprises re-training the first lead score model based at least in part on the first execution cycle data.

14. The method of claim 12 , wherein the determining the second lead scoring model comprises modifying a scoring parameter of the first lead score model.

15. The method of claim 12 , further comprising, after determining the second lead scoring model, determining that application of the second lead scoring model is permitted by a limitation rule.

16. The method of claim 12 , further comprising:

determining an average scrub rate for a second plurality of leads received during the analysis window; and

determining the scrub rate threshold based at least in part on the average scrub rate.

17. The method of claim 12 , further comprising:

determining a mean scrub rate of a plurality of historical scrub rates to generate the analysis window scrub rate; and

determining that the scrub rate threshold is more than two standard deviations higher than the mean scrub rate.

18. A non-transitory machine-readable medium comprising instructions which, when read by a machine, cause the machine to perform operations comprising:

receiving first execution cycle data comprising a first set of leads and a second set of leads, wherein the first set of leads comprises first lead information describing a first plurality of potential customers and the second set of leads comprises second lead information describing a second plurality of potential customers;

determining a first lead score for each of the plurality of potential customers in the set of first leads based at least in part on a first lead scoring model;

determining that the first lead scores for the plurality of potential customers in the set of first leads is greater than a first lead score threshold;

determining that a second lead score for the plurality of potential customers in the set of second leads is less than the first lead score threshold;

selecting a set of filtered leads including the first lead information from the first set of leads;

determining that a quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold, the analysis window data comprising a plurality of leads received during an analysis window time period;

determining a second lead scoring model based at least in part on the first execution cycle data;

receiving second execution cycle data comprising a third set of leads comprising third lead information describing a plurality of third potential customers;

determining a third lead score for each of the potential customer in the third set of leads based at least in part on the second lead scoring model including determining whether a guard-rail enforcer component controls the determined second lead scoring model;

determining that the third lead scores for the third leads are greater than the first lead score threshold;

selecting a second set of filtered leads including the third lead information from the third set of leads; and

adjust at least one of the scoring models on the lead scorer component based on the quality of leads determined by the plurality of lead scoring models;

wherein the determining that the quality of the first execution cycle data is different than a quality of analysis window data by more than a quality threshold comprises:

determining a scrub rate, wherein the scrub rate describes a portion of the first execution cycle data having lead scores greater than the first lead score threshold;

determine that the scrub rate is greater than an analysis window scrub rate by more than a scrub rate threshold; and

receiving lead score data describing lead scores for at least one lead received during an observation period and at least one lead received during the analysis window and not during the observation period.

Assignments (7)