Reinforcement learning with quantum oracle
A computing device is provided, including a processor configured to transmit, to a quantum coprocessor, instructions to encode a Markov decision process (MDP) model as a quantum oracle. The processor may be further configured to train a reinforcement learning model at least in part by transmitting a plurality of superposition queries to the quantum oracle encoded at the quantum coprocessor. Training the reinforcement learning model may further include receiving, from the quantum coprocessor, one or more measurement results in response to the plurality of superposition queries. Training the reinforcement learning model may further include updating a policy function of the reinforcement learning model based at least in part on the one or more measurement results.
1. A computing device comprising:
a processor configured to:
transmit, to a quantum coprocessor, instructions to encode a Markov decision process (MDP) model as a quantum oracle; and
train a reinforcement learning model at least in part by:
transmitting a plurality of superposition queries to the quantum oracle encoded at the quantum coprocessor, wherein:
a number of the superposition queries is proportional to an inverse of a target accuracy; and
the target accuracy is a predefined maximum distance between an optimal value estimate included in one or more measurement results and an optimal value approximated by the optimal value estimate;
performing one or more measurements at the quantum oracle as specified by the superposition queries;
receiving, from the quantum coprocessor, one or more measurement results of the one or more measurements in response to the plurality of superposition queries; and
updating a policy function of the reinforcement learning model based at least in part on the one or more measurement results.
2. The computing device of claim 1 , wherein the one or more measurement results include an estimated optimal Q-function, an estimated optimal value function, or an estimated optimal policy function for the reinforcement learning model.
3. The computing device of claim 2 , further comprising quantum random access memory (QRAM), wherein the quantum coprocessor is configured to compute the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function at least in part by making one or more memory calls to the QRAM.
4. The computing device of claim 2 , wherein the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function is computed at the quantum coprocessor via quantum amplitude estimation, quantum Monte Carlo mean estimation, or quantum minimum finding.
5. The computing device of claim 2 , wherein:
the one or more measurement results include the estimated optimal Q-function; and
the plurality of superposition queries includes a number of superposition queries proportional to a number of actions included in the MDP model.
6. The computing device of claim 2 , wherein:
the one or more measurement results include the estimated optimal value function or the estimated optimal policy function; and
the plurality of superposition queries includes a number of superposition queries proportional to a square root of a number of actions included in the MDP model.
7. The computing device of claim 1 , wherein the processor is configured to train the reinforcement learning model via Q-learning or approximate value iteration.
8. The computing device of claim 1 , wherein the one or more measurement results include, for the policy function of the reinforcement learning model:
an estimated value of a state distribution;
an estimated gradient of the estimated value of the state distribution; or
an estimated risk of obtaining a cumulative discounted reward below a reward threshold.
9. A method for use with a computing device, the method comprising:
transmitting, to a quantum coprocessor, instructions to encode a Markov decision process (MDP) model as a quantum oracle; and
training a reinforcement learning model at least in part by:
transmitting a plurality of superposition queries to the quantum oracle encoded at the quantum coprocessor, wherein:
a number of the superposition queries is proportional to an inverse of a target accuracy; and
the target accuracy is a predefined maximum distance between an optimal value estimate included in one or more measurement results and an optimal value approximated by the optimal value estimate;
performing one or more measurements at the quantum oracle as specified by the superposition queries;
receiving, from the quantum coprocessor, one or more measurement results of the one or more measurements in response to the plurality of superposition queries; and
updating a policy function of the reinforcement learning model based at least in part on the one or more measurement results.
10. The method of claim 9 , wherein the one or more measurement results include an estimated optimal Q-function, an estimated optimal value function, or an estimated optimal policy function for the reinforcement learning model.
11. The method of claim 10 , wherein the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function is computed at the quantum coprocessor at least in part by making one or more memory calls to quantum random access memory (QRAM).
12. The method of claim 10 , wherein the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function is computed at the quantum coprocessor via quantum amplitude estimation, quantum Monte Carlo mean estimation, or quantum minimum finding.
13. The method of claim 9 , wherein the reinforcement learning model is trained via Q-learning or approximate value iteration.
14. The method of claim 9 , wherein the one or more measurement results include, for the policy function of the reinforcement learning model:
an estimated value of a state distribution;
an estimated gradient of the estimated value of the state distribution; or
an estimated risk of obtaining a cumulative discounted reward below a reward threshold.
15. A quantum computing device comprising:
quantum random access memory (QRAM); and
a quantum processor configured to:
encode a Markov decision process (MDP) model as a quantum oracle, wherein the quantum oracle is a quantum analogue of the MDP model;
retrieve one or more vectors stored in the QRAM;
receive a plurality of superposition queries from a classical computing device, wherein:
a number of the superposition queries is proportional to an inverse of a target accuracy; and
the target accuracy is a predefined maximum distance between an optimal value estimate included in one or more measurement results and an optimal value approximated by the optimal value estimate;
at the quantum oracle, based at least in part on the one or more vectors and the plurality of superposition queries, compute an estimated optimal Q-function, an estimated optimal value function, or an estimated optimal policy for the MDP model; and
perform, by the classical computing device, one or more measurements at the quantum oracle to receive the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function,
wherein the classical computing device is configured to train a reinforcement learning model based at least in part on the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function.
16. The quantum computing device of claim 15 , wherein the one or more vectors stored in the QRAM include a reward vector of rewards included in the MDP model.
17. The quantum computing device of claim 15 , wherein the one or more vectors stored in the QRAM include a value function vector of respective values corresponding to states included in the MDP model.
18. The quantum computing device of claim 15 , wherein the quantum processor is configured to compute the estimated optimal Q-function, the estimated optimal value function, or the estimated optimal policy function via quantum amplitude estimation, quantum Monte Carlo mean estimation, or quantum minimum finding.
19. The quantum computing device of claim 15 , wherein the classical computing device is configured to train the reinforcement learning model via Q-learning or approximate value iteration.