IP Library › Granted Patent US 12,288,074
Granted Patent B2
US 12,288,074 · App. 16/261,092 · Granted Apr 29, 2025

Generating and providing proposed digital actions in high-dimensional action spaces using reinforcement learning models

Inventors: Yash Chandak (Amherst, MA); Georgios Theocharous (San Jose, CA)
Assignee: Adobe Inc.
G06F9/3818G06F9/4806G06N3/08G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,288,074
App. No.
16/261,092
Granted
Apr 29, 2025
Kind
B2
Abstract

The present disclosure relates to generating proposed digital actions in high-dimensional action spaces for client devices utilizing reinforcement learning models. For example, the disclosed systems can utilize a supervised machine learning model to train a latent representation decoder to determine proposed digital actions based on latent representations. Additionally, the disclosed systems can utilize a latent representation policy gradient model to train a state-based latent representation generation policy to generate latent representations based on the current state of client devices. Subsequently, the disclosed systems can identify the current state of a client device and a plurality of available actions, utilize the state-based latent representation generation policy to generate a latent representation based on the current state, and utilize the latent representation decoder to determine a proposed digital action from the plurality of available actions by analyzing the latent representation.

Claims (59)

1. A computer-implemented method for generating and providing digital action recommendations based on digital embeddings of action spaces, comprising:

training a latent representation decoder via supervised machine learning to learn parameters for generating proposed digital actions associated with a discrete action space from one or more latent representations within a continuous action space by using a supervised machine learning model to update the parameters of the latent representation decoder based on, for each training iteration, a training current state, a training next state, a predicted action generated by the latent representation decoder based on the training current state and the training next state, and a ground truth action comprising a digital action that causes a transition from the training current state to the training next state;

training, with the latent representation decoder, a latent representation generator via the supervised machine learning to learn additional parameters for generating latent representations within the continuous action space from training current states and training next states by using the supervised machine learning model to update the additional parameters of the latent representation generator based on, for each training iteration, the training current state, the training next state, the predicted action generated by the latent representation decoder based on a latent representation generated by the latent representation generator from the training current state and the training next state, and the ground truth action;

training a state-based latent representation generation policy via reinforcement learning to learn further parameters for generating additional latent representations within the continuous action space from current states by using a latent representation policy gradient model, that includes the latent representation decoder, to update the further parameters of the state-based latent representation generation policy based on a training reward;

identifying, for a client device, a current state and a plurality of available actions associated with the discrete action space; and

generating a proposed digital action from the plurality of available actions and the current state utilizing the state-based latent representation generation policy and the latent representation decoder.

2. The computer-implemented method of claim 1 , wherein the current state of the client device comprises at least one of digital content previously distributed to the client device or steps previously taken with respect to a process.

3. The computer-implemented method of claim 1 , wherein the proposed digital action comprises at least one of digital content to distribute or a step to take with respect to a process.

4. The computer-implemented method of claim 1 , wherein identifying the current state of the client device comprises determining an initial state of the client device from an initial distribution of states.

5. A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to:

identify a current state of a client device and a plurality of available actions; and

generate a proposed digital action for the client device from the plurality of available actions by:

generating, based on the current state of the client device and utilizing a state-based latent representation generation policy, a latent representation, wherein:

the state-based latent representation generation policy is trained to generate latent representations from current states and includes parameters learned via reinforcement learning by using a latent representation policy gradient model to update the parameters of the state-based latent representation generation policy based on a training reward;

the latent representation policy gradient model used for training the state-based latent representation generation policy via reinforcement learning comprises a latent representation decoder;

the latent representation decoder is trained to generate proposed digital actions from one or more latent representations and includes additional parameters learned via supervised machine learning by using a supervised machine learning model to update the additional parameters of the latent representation decoder based on, for each training iteration, a training current state, a training next state, a predicted action generated by the latent representation decoder from the training current state and the training next state, and a ground truth action comprising a digital action that causes a transition from the training current state to the training next state; and

the supervised machine learning model used for training the latent representation decoder via the supervised machine learning comprises a latent representation generator that generates additional latent representations from training current states and training next states and includes further parameters learned via the supervised machine learning by using the supervised machine learning model to update the further parameters of the latent representation generator based on, for each training iteration, the training current state, the training next state, the predicted action generated by the latent representation decoder based on an additional latent representation generated by the latent representation generator from the training current state and the training next state, and the ground truth action; and

determining the proposed digital action from the plurality of available actions by analyzing the latent representation utilizing the latent representation decoder.

6. The non-transitory computer readable storage medium of claim 5 , wherein:

the plurality of available actions comprises a plurality of discrete actions, and

the latent representation comprises a value from a set of continuous values corresponding to the plurality of discrete actions.

7. The non-transitory computer readable storage medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the client device to:

identify a next state of the client device and a second plurality of available actions corresponding to the client device upon execution of the proposed digital action; and

generate a second latent representation based on the next state of the client device utilizing the state-based latent representation generation policy.

8. The non-transitory computer readable storage medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the client device to determine a second proposed digital action from the second plurality of available actions by analyzing the second latent representation utilizing the latent representation decoder.

9. The non-transitory computer readable storage medium of claim 5 , wherein the proposed digital action comprises at least one of digital content to distribute or a step to take with respect to a process.

10. The non-transitory computer readable storage medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to train the latent representation decoder via the supervised machine learning by, for a first training iteration:

generating, utilizing the latent representation generator, a first latent representation based on a first training current state and a first training next state; and

generating, utilizing the latent representation decoder, a first predicted action based on the first latent representation.

11. The non-transitory computer readable storage medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computing device to train the latent representation decoder via the supervised machine learning by, for the first training iteration:

determining a loss based on comparing the first predicted action with a first ground truth action that includes a first digital action that causes a first transition from the first training current state to the first training next state; and

modifying parameters of the latent representation decoder based on the loss.

12. A system comprising:

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:

train a latent representation decoder via supervised machine learning to generate actions associated with a discrete action space from one or more latent representations within a continuous space by learning parameters of the latent representation decoder and additional parameters of a latent representation generator over a plurality of training iterations that each use a training current state, a training next state, a predicted action generated by the latent representation decoder from the training current state and the training next state, and a ground truth action comprising a digital action that causes a transition from the training current state to the training next state by, for a first training iteration:

utilizing a supervised machine learning model, that includes the latent representation decoder and the latent representation generator, to:

generate a first latent representation from a first training current state and a first training next state via the latent representation generator; and

generate a first predicted action from the first latent representation via the latent representation decoder; and

modifying the parameters of the latent representation decoder and the additional parameters of the latent representation generator based on comparing the first predicted action to a first ground truth action, the first ground truth action comprising a first digital action that causes a transition from the first training current state to the first training next state; and

utilize a latent representation policy gradient model that includes the latent representation decoder with the parameters modified via the supervised machine learning to train a state-based latent representation generation policy via reinforcement learning to generate, from current states and within the continuous space that corresponds to the discrete action space, latent representations for use in generating proposed digital actions by:

generating a second latent representation from a second training current state utilizing the state-based latent representation generation policy;

utilizing the latent representation decoder trained via the supervised machine learning to generate a proposed action from the second latent representation; and

modifying the state-based latent representation generation policy using a policy gradient, based on a second training next state and a training reward resulting from the proposed action.

13. The system of claim 12 , wherein:

the first training current state and the second training current state are a same training current state;

the first training next state and the second training next state are a same training next state; and

the ground truth action and the proposed action are a same action.

14. The system of claim 12 , wherein modifying the state-based latent representation generation policy using the policy gradient comprises modifying further parameters of the state-based latent representation generation policy using the policy gradient without modifying the parameters of the latent representation decoder.

15. The system of claim 12 , wherein modifying the additional parameters of the latent representation generator comprises modifying the additional parameters using a loss function, based on comparing the first predicted action to the first ground truth action, the first ground truth action corresponding to the first training current state and the first training next state.

16. The system of claim 14 , wherein the latent representation generator comprises a neural network.

17. The system of claim 12 , further comprising instructions that, when executed by the at least one processor, cause the system to utilize the latent representation policy gradient model to train the state-based latent representation generation policy by determining the second training next state and the training reward based on execution of the proposed action.

18. The system of claim 17 ,

further comprising instructions that, when executed by the at least one processor, cause the system to provide the proposed action to a client device,

wherein determining the second training next state and the training reward based on execution of the proposed action comprises determining the second training next state and the training reward based on execution of the proposed action by the client device.

19. The system of claim 12 , wherein modifying the parameters of the latent representation decoder based on comparing the first predicted action to the first ground truth action comprises modifying the parameters of the latent representation decoder using a loss function.

20. The system of claim 12 , wherein:

the second latent representation comprises a value from a set of continuous values corresponding to a plurality of discrete actions associated with the second training current state, and

the plurality of discrete actions associated with the second training current state comprises the proposed action.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 29, 2019
From: CHANDAK, YASH; THEOCHAROUS, GEORGIOS
To: ADOBE INC.
Reel/Frame 048169/0867 →
Continuity (1)
Related Publication 20200241878A1 · Jul 30, 2020
References Cited (41)
US 20150143395A1 · Reisman · 2015 [cited by examiner]
US 20190005021A1 · Miller · 2019 [cited by examiner]
US 20190251612A1 · Fang · 2019 [cited by examiner]
WO WO2018211140A1 · 2018 [cited by examiner]
Lillicrap, Timothy P. et al. “Continuous control with deep reinforcement learning” Google Deepmind [Version 5 published Feb. 29, 2016] [Retrieved Dec. 2022] <URL: https://doi.org/10.48550/arXiv.1509.02971> (Year: 2016). [cited by examiner]
Kearns, Michael, and Daphne Koller. “Efficient reinforcement learning in factored MDPs.” IJCAI. vol. 16. 1999. (Year: 1999). [cited by examiner]
Ghosh, Dibya et al. “Learning Actionable Representations with Goal-Conditioned Policies” UCB [Version 1 published Nov. 19, 2018] [ Retrieved Dec. 2022] <URL: https://doi.org/10.48550/arXiv.1811.07819> (Year: 2018). [cited by examiner]
Nair, Ashvin et al. “Visual Reinforcement Learning with Imagined Goals” NeuralIPS'18 [Published Dec. 2018] [Retrieved May 2023] <URL: https://doi.org/10.48550/arXiv.1807.04742> (Year: 2018). [cited by examiner]
Shelhamer, Evan et al. “Loss is its own Reward: Self-Supervision for Reinforcement Learning” arXiv. [Published Mar. 2017] [Retrieved May 2023] <URL: https://doi.org/10.48550/arXiv.1612.07307> (Year: 2017). [cited by examiner]
Authors: Zhuang et al Title: Supervised Representation Learning: Transfer Learning with Deep Autoencoders Date: 2015 (Year: 2015). [cited by examiner]
Akata, Z.; Perronnin, F.; Harchaoui, Z.; and Schmid, C. 2016. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence 38(7):1425-1438. [cited by applicant]
Barreto, A.; Dabney, W.; Munos, R.; Hunt, J. J.; Schaul, T.; van Hasselt, H. P.; and Silver, D. 2017. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, 4055… [cited by applicant]
Dayan, P. 1993. Improving generalization for temporal difference learning: The successor representation. Neural Com-putation 5(4):613-624. [cited by applicant]
Dulac-Arnold, G.; Evans, R.; van Hasselt, H.; Sunehag, P.; Lillicrap, T.; Hunt, J.; Mann, T.; Weber, T.; Degris, T.; and Coppin, B. 2015. Deep reinforcement learning in large dis-crete action spaces. arXiv preprint arXi… [cited by applicant]
Flash, T., and Hochner, B. 2005. Motor primitives in vertebrates and invertebrates. Current opinion in neurobiology 15(6):660-666. [cited by applicant]
Ijspeert, A. J.; Nakanishi, J.; and Schaal, S. 2003. Learning attractor landscapes for learning motor primitives. In Advances in neural information processing systems, 1547-1554. [cited by applicant]
Jaderberg, M.; Mnih, V.; Czarnecki, W. M.; Schaul, T.; Leibo, J. Z.; Silver, D.; and Kavukcuoglu, K. 2016. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397. [cited by applicant]
Jing, J.; Cropper, E. C.; Hurwitz, I.; and Weiss, K. R. 2004. The construction of movement with behavior-specific and behavior-independent modules. Journal of Neuroscience 24(28):6315-6325. [cited by applicant]
Kober, J., and Peters, J. 2009a. Learning motor primitives for robotics. In Robotics and Automation, 2009. ICRA'09. IEEE International Conference on, 2112-2118. IEEE. [cited by applicant]
Kober, J., and Peters, J. R. 2009b. Policy search for motor primitives in robotics. In Advances in neural information processing systems, 849-856. [cited by applicant]
Konidaris, G.; Osentoski, S.; and Thomas, P. S. 2011. Value function approximation in reinforcement learning using the fourier basis. In AAAI, vol. 6, 7. [cited by applicant]
Mahadevan, S., and Maggioni, M. 2007. Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research 8(Oct.): 2169-2231. [cited by applicant]
Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 3111-3119. [cited by applicant]
Osentoski, S., and Mahadevan, S. 2007. Learning state-action basis functions for hierarchical mdps. In Proceedings of the 24th international conference on Machine learning, 705-712. ACM. [cited by applicant]
Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), vol. 2017. [cited by applicant]
Pazis, J., and Parr, R. 2011. Generalized value functions for large action sets. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1185-1192. [cited by applicant]
Sallans, B., and Hinton, G. E. 2004. Reinforcement learning with factored states and actions. Journal of Machine Learning Research 5(Aug.):1063-1088. [cited by applicant]
Schaal, S. 2006. Dynamic movement primitives—a framework for motor control in humans and humanoid robotics. In Adaptive motion of animals and machines. Springer. 261-280. [cited by applicant]
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [cited by applicant]
Shani, G.; Heckerman, D.; and Brafman, R. I. 2005. An mdp-based recommender system. Journal of Machine Learning Research 6(Sep.):1265-1295. [cited by applicant]
Sharma, S.; Suresh, A.; Ramesh, R.; and Ravindran, B. 2017. Learning to factor policies and action-value functions: Factored action space representations for deep reinforcement learning. arXiv preprint arXiv:1705.07269. [cited by applicant]
Shelhamer, E.; Mahmoudieh, P.; Argus, M.; and Darrell, T. 2016. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307. [cited by applicant]
Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; and Riedmiller, M. 2014. Deterministic policy gradient algorithms. In ICML. [cited by applicant]
Sutton, R. S.; McAllester, D. A.; Singh, S. P.; and Mansour, Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, 1057-1063. [cited by applicant]
Theocharous, G.; Thomas, P. S.; and Ghavamzadeh, M. 2015. Ad recommendation systems for life-time value optimization. In Proceedings of the 24th International Conference on World Wide Web, 1305-1310. ACM. [cited by applicant]
Thomas, P. S., and Barto, A. G. 2011. Conjugate markov decision processes. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 137-144. [cited by applicant]
Thomas, P. S., and Barto, A. G. 2012. Motor primitive discovery. In Development and Learning and Epigenetic Robotics (ICDL), 2012 IEEE International Conference on, 1-8. IEEE. [cited by applicant]
Thomas, P. S. 2011. Policy gradient coagent networks. In Advances in Neural Information Processing Systems, 1944-1952. [cited by applicant]
Thomas, P. 2014. Bias in natural actor-critic algorithms. In International Conference on Machine Learning, 441-448. [cited by applicant]
Todorov, E., and Ghahramani, Z. 2003. Unsupervised learning of sensory-motor primitives. In Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE, vol. 2,… [cited by applicant]
Van Hasselt, H., and Wiering, M. A. 2009. Using continuous action spaces to solve discrete problems. In Neural Networks, 2009. IJCNN 2009. International Joint Conference on, 1149-1156. IEEE. [cited by applicant]