Memory device with reinforcement learning with Q-learning acceleration
The present disclosure generally relates to data storage devices, such as solid state drives. A data storage device includes a controller, one or more volatile memory locations, and one or more non-volatile memory locations. Computations, including reinforcement learning algorithms, may be completed by the controller using the one or more non-volatile memory locations. Data associated with reinforcement learning is stored in a table on one or more planes of the non-volatile memory, where the results from the computations update the table with the relevant values. The data in the table are aligned to one or more wordlines, such that sensing the wordline senses all the data stored in the table.
1. A data storage device, comprising:
a non-volatile memory (NVM) device; and
a controller coupled to the NVM device, wherein the controller is configured to:
receive Q table data from the NVM device;
execute a Q-learning algorithm to create updated Q table data;
write the updated Q table data to a first Q table or a second Q table in the NVM device, wherein:
all actions associated with a state of one or more states of the first Q table and the second Q table are generated on a same wordline of the NVM device; and
on the same wordline, each action of the state of the first Q table and each action of the state of the second Q table are alternating; and
read the same wordline of the NVM device, wherein reading the same wordline senses all actions associated with the state of the one or more states of the first Q table and the second Q table.
2. The data storage device of claim 1 , wherein the Q table comprises a first Q table and the second Q table are arranged on a first wordline of a first page of a first plane of a first die of the NVM device.
3. The data storage device of claim 1 , wherein the first Q table is arranged on a first wordline of a first page of a first plane of a first die and the second Q table is arranged on a first wordline of a first page of a second plane of the first die.
4. The data storage device of claim 1 , wherein executing a Q-learning algorithm to create the updated Q table data comprises executing the following formula:
Define a *=argmax a Q A ( s′,a )
Q A ( s,a )← Q A ( s,a )+α( s,a )[ r+γQ B ( s′,a *)− Q A ( s,a )]
where a* is the optimal action to be taken in the next step according to the first Q table (Q A ), where a is the action chosen at the current step, where s is the current state, where s′ is the next state, where r is the immediate reward received in the current step, where γ is a discount factor, and where α(s,a) is a step size for the current state and the current action.
5. The data storage device of claim 4 , wherein executing a Q-learning algorithm to create the updated Q table data additionally comprises executing the following formula:
Define b *=argmax b Q B ( s′,a )
Q B ( s,a )← Q B ( s,a )+α( s,a )[ r+γQ A ( s′,a *)− Q B ( s,a )]
wherein a* is the optimal action to be taken in the next step according to the second Q table (Q B ), wherein a is the action chosen at the current step, wherein s is the current state, wherein s′ is the next state, wherein r is the immediate reward received in the current step, wherein γ is a discount factor, and wherein α(s,a) is a step size for the current state and the current action.
6. The storage device of claim 5 , wherein the controller is further configured to:
advance to a second state of both the first Q table and the second Q table; and
read the data of the second state from both the first Q table and the second Q table.
7. A data storage device, comprising:
a non-volatile memory (NVM) device, wherein the NVM device is configured to:
read a wordline of the NVM device, wherein reading the wordline senses all actions associated with a state of one or more states of a first Q table and a second Q table;
execute a Q-learning algorithm to create updated Q table data; and
write the updated Q table data to the first Q table or the second Q table in the NVM device, wherein:
all actions associated with a state of one or more states of the first Q table and the second Q table are generated on a same wordline of the NVM device; and
on the same wordline, each action of the state of the first Q table and each action of the state of the second Q table are alternating; and
a controller coupled to the NVM device.
8. The data storage device of claim 7 , wherein the first or second Q table data and the updated Q table data are stored on a same page of the NVM device.
9. The data storage device of claim 7 , wherein the NVM device comprises CMOS by the Array, and wherein executing the Q-learning algorithm to create the updated Q table data is performed in the CMOS by the Array.
10. The data storage device of claim 7 , wherein the NVM device comprises CMOS under the Array (CuA), and wherein executing the Q-learning algorithm to create the updated Q table data is performed by the CuA.
11. The data storage device of claim 7 , wherein the NVM device comprises CMOS bounded Array (CbA), and wherein executing the Q-learning algorithm to create the updated Q table data is performed by the CbA.
12. A data storage device, comprising:
a non-volatile memory (NVM) device;
means to generate one or more Q tables, wherein:
each of the one or more Q tables comprises one or more states,
a first state of a first Q table and a first state of a second Q table of the one or more Q tables are generated on a wordline of the NVM device,
all actions associated with the first state of the first Q table and the first state of the second Q table are generated on the wordline, and
on the wordline, each action of the first state of the first Q table and each action of the first state of the second Q table are alternating;
means to sense the wordline, wherein sensing the wordline senses all the data for the first states of the first Q table and the second Q table;
means to perform a max argument calculation for the one or more of the Q tables;
means to update the one or more Q tables with the results of the max argument calculation; and
means to advance to the next state of the one or more Q tables.
13. The data storage device of claim 12 , wherein the means to update the one or more Q tables occurs on the same page of a first die.
14. The data storage device of claim 12 , wherein the first Q table of the one or more Q tables is generated on a first plane of the NVM device and the second Q table of the one or more Q tables is generated on a second plane of the NVM device.
15. The data storage device of claim 12 , wherein the structure of the one or more Q tables is a two dimensional structure stored in the NVM device, wherein the two dimensional structure comprises a size, a location, and a resolution of each entry for the one or more Q tables.
16. The data storage device of claim 12 , wherein the means to perform a max argument calculation is distinct from the NVM device.
17. The data storage device of claim 12 , wherein the means to perform a max argument calculation is an integral part of the NVM device.
18. The data storage device of claim 12 , wherein the means to perform the max argument calculation performs the following calculation:
Q ( s t ,a t )← Q ( s t ,a t )+α[ r t+1 +γargmaxQ( s t+1 ,a )− Q ( s t ,a t )]
wherein s is the current state, wherein a is the current action, wherein Q(s,a) is the max Q table value for all current states and actions, wherein r is the reward for the current action for the current state, and wherein γ is a discount factor.