Ad exchange bid optimization with reinforcement learning

1. A system for training a bidding model by reinforcement learning using first and second neural networks, the system configured to:

perform operations, in response to an available inventory from a publisher relayed through a real time bid server, including processing a bid on the available inventory, the processing including:

sending the bid to the real time bid server;

receiving a bid result in response to the bid;

storing state data including a sequence of bids sent to the real time bid server, the bid result, and a response rate for the available inventory;

using the first neural network, determining a plurality of target action Q-values based on the state data, the plurality of target action Q-values including at least one Q-value for each possible action at a current state of the bid server;

selecting an action based on a maximum target action Q-value;

using a second machine learning model, determining a target Q-value for the selected action based on the state data and experience data, the experience data including the selected action and a reward earned for the selected action;

training the first neural network to update the plurality of target action Q-values based on the target Q-value, the training using a stochastic gradient descent;

calculating a plurality of losses; and

back propagating the losses through the bidding model.