Ad exchange bid optimization with reinforcement learning

1. A machine learning system comprising:

one or more processors; and

a memory storing instructions that, when executed by at least one processor in the one or more processors, cause the at least one processor to perform operations comprising at least:

receive a request to deploy a piece of content to available inventory;

receive a bid result for a bid on the available inventory, the bid submitted to a real time bid server in response to the request;

store state data including a sequence of bids sent to the real time bid server, the bid result, and a response rate for the available inventory;

train a bidding model by:

using a first neural network, determining a plurality of action Q-values based on the state data, the plurality of action Q-values including at least one Q-value for each possible action at a current state of the bid server;

selecting an action based on a maximum action Q-value;

using a second machine learning model, determining a target Q-value for the selected action based on the state data and experience data, the experience data including the selected action and a reward earned for the selected action;

training the first neural network to update the plurality of action Q-values based on the target Q-value, the training using a stochastic gradient descent;

determine a bid action using the trained bidding model; and

submit a new bid on the available inventory to the real time bid server based on the bid action.

2. The system of claim 1 , wherein each action Q-value corresponds to a value loss or value gain achieved by performing an action that transitions the bid server from an initial state to an end state.

3. The system of claim 1 , wherein the training using a stochastic gradient descent further comprises using an adaptive learning rate that decreases a step size over time, the step size used to adjust multiple weights of the first neural network.

4. The system of claim 1 , wherein the bidding model is further trained by copying multiple weights from the second neural network to the first neural network after a predetermined number of training steps; and wherein the operations further comprise:

training a new set of weights for the second neural network based on outputs from the first neural network determined using the copied weights.

5. The system of claim 1 , wherein the bidding model is further trained by calculating a plurality of losses; and wherein the operations further comprise:

backpropagating the losses through the bidding model.

6. The system of claim 1 , wherein the bid action optimizes at least one of a bid timing and a bid amount of the new bid.

7. The system of claim 1 , wherein the operations further comprise dynamically adjust allocation data of the request based on the bid action.

8. The system of claim 7 , wherein the allocation data includes a number of units of the piece of content to be deployed among multiple platforms; and wherein

the operations further comprise modifying the number of units based on the adjusted allocation data.

9. The system of claim 7 , wherein the allocation data includes a budget value determined based on a number of units of the piece of content allocated among multiple platforms; and wherein

the operations further comprise deploying the piece of content on at least one of the multiple platforms using the adjusted allocation data; and

collecting delivery and performance information for the piece of content on the at least one of the multiple platforms.

10. The system of claim 9 , wherein the operations further comprise determining an adjusted budget value based on the delivery and performance information; and

dynamically reallocating a budget amount among the multiple platforms based on the adjusted budget value.

11. A method of allocating data across multiple platforms, the method comprising:

receiving a request to deploy a piece of content to available inventory;

receiving a bid result for a bid on the available inventory, the bid submitted to a real time bid server in response to the request;

storing state data including a sequence of bids sent to the real time bid server, the bid result, and a response rate for the available inventory;

training a bidding model by:

using a first neural network, determining a plurality of action Q-values based on the state data, the plurality of action Q-values including at least one Q-value for each possible action at a current state of the bid server;

selecting an action based on a maximum action Q-value;

using a second machine learning model, determining a target Q-value for the selected action based on the state data and experience data, the experience data including the selected action and a reward earned for the selected action;

training the first neural network to update the plurality of action Q-values based on the target Q-value, the training using a stochastic gradient descent;

determining a bid action using the trained bidding model; and

submitting a new bid on the available inventory to the real time bid server based on the bid action.

12. The method of claim 11 , wherein each action Q-value corresponds to a value loss or value gain achieved by performing an action that transitions the bid server from an initial state to an end state.

13. The method of claim 11 , wherein the training using a stochastic gradient descent further comprises using an adaptive learning rate that decreases a step size over time, the step size used to adjust multiple weights of the first neural network.

14. The method of claim 11 , wherein training the bidding model further comprises:

copying multiple weights from the second neural network to the first neural network after a predetermined number of training steps; and

training a new set of weights for the second neural network based on outputs from the first neural network determined using the copied weights.

15. The method of claim 11 , wherein training the bidding model further comprises:

calculating a plurality of losses; and

backpropagating the losses through the bidding model.

16. The method of claim 11 , wherein the bid action optimizes at least one of a bid timing and a bid amount of the new bid.

17. The method of claim 11 , further comprising dynamically adjusting allocation data of the request based on the bid action.

18. The method of claim 17 , wherein the allocation data includes a number of units of the piece of content to be deployed among multiple platforms; and wherein

the method further comprises modifying the number of units based on the adjusted allocation data.

19. The method of claim 17 , wherein the allocation data includes a budget value determined based on a number of units of the piece of content allocated among multiple platforms; and wherein

the method further comprises deploying the piece of content on at least one of the multiple platforms using the adjusted allocation data; and

collecting delivery and performance information for the piece of content on the at least one of the multiple platforms.

20. The method of claim 19 , further comprising determining an adjusted budget value based on the delivery and performance information; and

dynamically reallocating a budget amount among the multiple platforms based on the adjusted budget value.