SYSTEMS AND METHODS FOR PERFORMANCE ADVERTISING SMART OPTIMIZATIONS
Systems and methods applicable to generating management decisions for online advertising. Machine learning models, including reinforcement learning-based machine learning models, can be utilized in making various advertising management decisions.
1 . A computer-implemented method, comprising:
providing, by a computing system, to a reinforcement learning-based machine learning model, observations received from an online advertisement environment; and
receiving, by the computing system, from the reinforcement learning-based machine learning model, one or more budget allocation actions,
wherein training of the reinforcement learning-based machine learning model seeks a policy that minimizes penalty reward issued by the online advertisement environment.
2 . The computer-implemented method of claim 1 , wherein the observations received from the online advertisement environment comprise one or more of spend rate, cost per action, pacing, cost per mile, or conversion rate.
3 . The computer-implemented method of claim 1 , wherein the reinforcement learning-based machine learning model includes an actor and a critic.
4 . The computer-implemented method of claim 1 , wherein the reinforcement learning-based machine learning model is implemented via a multi-arm bandit-based actor-critic algorithm, A 2 C, or A 3 C.
5 . The computer-implemented method of claim 1 , wherein the budget allocation actions specify, for each of multiple ad entities, a budget allocation.
6 . The computer-implemented method of claim 1 , wherein the penalty reward comprises one or more of a cost per action penalty or a spend penalty.
7 . A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform the computer-implemented method of claim 1 .
8 . A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform the computer-implemented method of claim 1 .
9 . A computer-implemented method, comprising:
providing, by a computing system, to a reinforcement learning-based machine learning model, observations received from an online advertisement auction house environment; and
receiving, by the computing system, from the reinforcement learning-based machine learning model, one or more bid update actions,
wherein training of the reinforcement learning-based machine learning model seeks a policy that maximizes estimated cost per action-based reward.
10 . The computer-implemented method of claim 9 , wherein the observations received from the online advertisement environment comprise one or more of conversion rate, spend rate, cost per action, and cost per mile.
11 . The computer-implemented method of claim 9 , wherein the reinforcement learning-based machine learning model includes an actor and a critic.
12 . The computer-implemented method of claim 9 , wherein the reinforcement learning-based machine learning model is implemented via A 2 C or A 3 C.
13 . The computer-implemented method of claim 9 , wherein the estimated cost per action-based reward is implemented via a reward function that:
utilizes, under a circumstance where an estimated cost per action is greater than a target cost per action, deviation of the estimated cost per action from the target cost per action, and
utilizes, under a circumstance where the estimated cost per action is less than the target cost per action, deviation of estimated pacing from desired pacing.
14 . The computer-implemented method of claim 9 , further comprising:
utilizing, by the computing system, bid multipliers to account for incrementality differences across ad entities.
15 . A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform the computer-implemented method of claim 9 .
16 . A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform the computer-implemented method of claim 9 .
17 . A computer-implemented method, comprising:
providing, by a computing system, to a reinforcement learning-based machine learning model, observations received from an online advertisement auction house environment; and
receiving, by the computing system, from the reinforcement learning-based machine learning model, one or more bid multiplier actions,
wherein training of the reinforcement learning-based machine learning model seeks a policy that maximizes conversion reward issued by the online advertisement auction house environment.
18 . The computer-implemented method of claim 17 , wherein the observations received from the online advertisement auction house environment comprise one or more of audience segment spend rates or audience segment conversion rates.
19 . The computer-implemented method of claim 17 , wherein the reinforcement learning-based machine learning model is implemented via A 2 C or A 3 C.
20 . The computer-implemented method of claim 17 , wherein the bid multiplier actions are applied to bid update actions generated by a further reinforcement learning-based machine learning model.
21 . A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform the computer-implemented method of claim 17 .
22 . A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform the computer-implemented method of claim 17 .