IP Library › Granted Patent US 12,705,477
Granted Patent B2
US 12,705,477 · App. 17/180,682 · Granted Aug 11, 2026

Learning policies using sparse and underspecified rewards

Inventors: Rishabh Agarwal (Montreal, CA); Chen Liang (Stanford, CA); Dale Eric Schuurmans (Mountain View, CA); Mohammad Norouzi (Richmond Hill, CA)
Assignee: Google LLC
G06N3/08G06F18/2148G06F18/217G06N3/092
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,705,477
App. No.
17/180,682
Filed
Feb 19, 2021
Granted
Aug 11, 2026
Kind
B2
Art Unit
2144
USPC
706/25
Abstract

Methods and systems for learning policies using sparse and underspecified rewards. One of the methods includes training the policy jointly with an auxiliary reward function having a plurality of auxiliary reward parameters, the auxiliary reward function being configured to map, in accordance with the auxiliary reward parameters, trajectory features of at least a trajectory to an auxiliary reward value that indicates how well the trajectory performed a task in response to a context input.

Claims (58)

1 . A method of learning a policy having a plurality of policy parameters configured to receive a context input for an agent to perform a task and to generate, from the context input and in accordance with the policy parameters, a trajectory of one or more actions for performing the task, wherein the one or more actions comprise control inputs for controlling the agent, the method comprising:

obtaining a plurality of training context inputs and a plurality of validation context inputs;

generating successful trajectory data that identifies a plurality of successful trajectories, each successful trajectory corresponding to a respective context input from either the training context inputs or the validation context inputs and being a trajectory for which a received task reward value indicates that the task was successfully completed by performing the trajectory in response to the corresponding context input; and

training the policy jointly with an auxiliary reward function having a plurality of auxiliary reward parameters, the auxiliary reward function being configured to map, in accordance with the auxiliary reward parameters, trajectory features of at least a trajectory to an auxiliary reward value that indicates how well the trajectory performed the task in response to the context input, and the training comprising repeatedly performing at least the following operations:

obtaining, from the successful trajectory data, a plurality of training successful trajectories, wherein each training successful trajectory is a successful trajectory that corresponds to a respective one of the training context inputs;

determining, using the auxiliary reward function and in accordance with current values of the auxiliary reward parameters, a respective auxiliary reward for each training successful trajectory;

training the policy to update current values of the policy parameters using the auxiliary rewards for the training successful trajectories;

obtaining, from the successful trajectory data, a plurality of validation successful trajectories, wherein each validation successful trajectory is a successful trajectory that corresponds to a respective one of the validation context inputs; and

training the auxiliary reward function on an objective that measures, for each validation successful trajectory, a probability assigned to the validation successful trajectory by outputs of the policy generated by processing the corresponding validation context input in accordance with the updated values of the policy parameters to update the current values of the auxiliary reward parameters, comprising computing gradients of the objective with respect to the auxiliary reward parameters using gradients of the objective with respect to the updated values of the policy parameters.

2 . The method of claim 1 , the following operations further comprising:

obtaining a training mini-batch of training context inputs and a validation mini-batch of validation context inputs;

for each context input in the training and validation mini-batches, generating a plurality of exploratory trajectories using the policy and in accordance with the current values of the policy parameters;

obtaining a respective task reward value for each of the exploratory trajectories that indicates whether the task was successfully completed by performing the exploratory trajectory; and

adding, to the successful trajectory data, any exploratory trajectories for which the task reward indicates that the task was successfully completed.

3 . The method of claim 1 , wherein the auxiliary reward function is a linear function that computes the auxiliary reward value as a linear combination of the task features based on the auxiliary reward parameters.

4 . The method of claim 1 , wherein the auxiliary reward function is a neural network that receives the task features as input and processes the task features in accordance with the auxiliary reward parameters to generate the auxiliary task reward.

5 . The method of claim 1 , wherein generating successful trajectory data that identifies a plurality of successful trajectories comprises:

training the policy starting from initial values of the policy parameters to generate first values of the policy parameters using only task rewards; and

periodically during the training using the policy to identify successful trajectories.

6 . The method of claim 5 , wherein training the policy starting from initial values of the policy parameters to generate first values of the policy parameters using only task rewards comprises training the policy to optimize an Iterative Maximum Likelihood (IML) objective.

7 . The method of claim 5 , wherein training the policy jointly with the auxiliary reward function comprises training the policy starting from the first values of the policy parameters.

8 . The method of claim 1 , wherein generating successful trajectory data that identifies a plurality of successful trajectories comprises:

identifying at least a subset of the trajectories in the successful trajectory data using random search.

9 . The method of claim 1 , wherein updating the current values of the policy parameters comprises updating the current values of the policy parameters to optimize a Memory Augmented Policy Optimization (MAPO) objective that is based on the auxiliary rewards.

10 . The method of claim 1 , wherein the objective measures, for each validation successful trajectory, a product of the received task reward value for the validation successful trajectory and a probability assigned to the validation successful trajectory by outputs of the policy generated by processing the corresponding validation context input in accordance with the updated values of the policy parameters to update the current values of the auxiliary reward parameters.

11 . A system for learning a policy having a plurality of policy parameters and configured to receive a context input for an agent to perform a task and to generate, from the context input and in accordance with the policy parameters, a trajectory of one or more actions for performing the task, wherein the one or more actions comprise control inputs for controlling the agent, the system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining a plurality of training context inputs and a plurality of validation context inputs;

generating successful trajectory data that identifies a plurality of successful trajectories, each successful trajectory corresponding to a respective context input from either the training context inputs or the validation context inputs and being a trajectory for which a received task reward value indicates that the task was successfully completed by performing the trajectory in response to the corresponding context input; and

training the policy jointly with an auxiliary reward function having a plurality of auxiliary reward parameters, the auxiliary reward function being configured to map, in accordance with the auxiliary reward parameters, trajectory features of at least a trajectory to an auxiliary reward value that indicates how well the trajectory performed the task in response to the context input, and the training comprising repeatedly performing at least the following operations:

obtaining, from the successful trajectory data, a plurality of training successful trajectories, wherein each training successful trajectory is a successful trajectory that corresponds to a respective one of the training context inputs;

determining, using the auxiliary reward function and in accordance with current values of the auxiliary reward parameters, a respective auxiliary reward for each training successful trajectory;

training the policy to update current values of the policy parameters using the auxiliary rewards for the training successful trajectories;

obtaining, from the successful trajectory data, a plurality of validation successful trajectories, wherein each validation successful trajectory is a successful trajectory that corresponds to a respective one of the validation context inputs; and

training the auxiliary reward function on an objective that measures, for each validation successful trajectory, a probability assigned to the validation successful trajectory by outputs of the policy generated by processing the corresponding validation context input in accordance with the updated values of the policy parameters to update the current values of the auxiliary reward parameters, comprising computing gradients of the objective with respect to the auxiliary reward parameters using gradients of the objective with respect to the updated values of the policy parameters.

12 . The system of claim 11 , the following operations further comprising:

obtaining a training mini-batch of training context inputs and a validation mini-batch of validation context inputs;

for each context input in the training and validation mini-batches, generating a plurality of exploratory trajectories using the policy and in accordance with the current values of the policy parameters;

obtaining a respective task reward value for each of the exploratory trajectories that indicates whether the task was successfully completed by performing the exploratory trajectory; and

adding, to the successful trajectory data, any exploratory trajectories for which the task reward indicates that the task was successfully completed.

13 . The system of claim 11 , wherein the auxiliary reward function is a linear function that computes the auxiliary reward value as a linear combination of the task features based on the auxiliary reward parameters.

14 . The system of claim 11 , wherein the auxiliary reward function is a neural network that receives the task features as input and processes the task features in accordance with the auxiliary reward parameters to generate the auxiliary task reward.

15 . The system of claim 11 , wherein generating successful trajectory data that identifies a plurality of successful trajectories comprises:

training the policy starting from initial values of the policy parameters to generate first values of the policy parameters using only task rewards; and

periodically during the training using the policy to identify successful trajectories.

16 . The system of claim 15 , wherein training the policy starting from initial values of the policy parameters to generate first values of the policy parameters using only task rewards comprises training the policy to optimize an Iterative Maximum Likelihood (IML) objective.

17 . The system of claim 15 , wherein training the policy jointly with the auxiliary reward function comprises training the policy starting from the first values of the policy parameters.

18 . The system of claim 11 , wherein generating successful trajectory data that identifies a plurality of successful trajectories comprises:

identifying at least a subset of the trajectories in the successful trajectory data using random search.

19 . The system of claim 11 , wherein updating the current values of the policy parameters comprises updating the current values of the policy parameters to optimize a Memory Augmented Policy Optimization (MAPO) objective that is based on the auxiliary rewards.

20 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for learning a policy having a plurality of policy parameters and configured to receive a context input for an agent to perform a task and to generate, from the context input and in accordance with the policy parameters, a trajectory of one or more actions for performing the task, wherein the one or more actions comprise control inputs for controlling the agent, the operations comprising:

obtaining a plurality of training context inputs and a plurality of validation context inputs;

generating successful trajectory data that identifies a plurality of successful trajectories, each successful trajectory corresponding to a respective context input from either the training context inputs or the validation context inputs and being a trajectory for which a received task reward value indicates that the task was successfully completed by performing the trajectory in response to the corresponding context input; and

training the policy jointly with an auxiliary reward function having a plurality of auxiliary reward parameters, the auxiliary reward function being configured to map, in accordance with the auxiliary reward parameters, trajectory features of at least a trajectory to an auxiliary reward value that indicates how well the trajectory performed the task in response to the context input, and the training comprising repeatedly performing at least the following operations:

obtaining, from the successful trajectory data, a plurality of training successful trajectories, wherein each training successful trajectory is a successful trajectory that corresponds to a respective one of the training context inputs;

determining, using the auxiliary reward function and in accordance with current values of the auxiliary reward parameters, a respective auxiliary reward for each training successful trajectory;

training the policy to update current values of the policy parameters using the auxiliary rewards for the training successful trajectories;

obtaining, from the successful trajectory data, a plurality of validation successful trajectories, wherein each validation successful trajectory is a successful trajectory that corresponds to a respective one of the validation context inputs; and

training the auxiliary reward function on an objective that measures, for each validation successful trajectory, a probability assigned to the validation successful trajectory by outputs of the policy generated by processing the corresponding validation context input in accordance with the updated values of the policy parameters to update the current values of the auxiliary reward parameters, comprising computing gradients of the objective with respect to the auxiliary reward parameters using gradients of the objective with respect to the updated values of the policy parameters.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 2, 2021
From: AGARWAL, RISHABH; LIANG, CHEN; SCHUURMANS, DALE ERIC; NOROUZI, MOHAMMAD
To: GOOGLE LLC
Reel/Frame 055452/0907 →
Continuity (2)
Provisional Application 62978793 · Feb 19, 2020
Related Publication 20210256313A1 · Aug 19, 2021
References Cited (102)
US 11074480B2 · Kim · 2021 [cited by examiner]
US 11429854B2 · El-Yaniv · 2022 [cited by examiner]
US 11449556B2 · Jawagal · 2022 [cited by examiner]
US 11477243B2 · Rao · 2022 [cited by examiner]
US 11562829B2 · Wei · 2023 [cited by examiner]
US 11568236B2 · Yu · 2023 [cited by examiner]
US 11580445B2 · Liu · 2023 [cited by examiner]
US 11620572B2 · Trott · 2023 [cited by examiner]
US 11651208B2 · Xu · 2023 [cited by examiner]
US 12293283B2 · Zheng · 2025 [cited by examiner]
US 20180157973A1 · El-Yaniv · 2018 [cited by examiner]
US 20190228309A1 · Yu · 2019 [cited by examiner]
US 20200175364A1 · Xu · 2020 [cited by examiner]
US 20200250486A1 · Kim · 2020 [cited by examiner]
US 20200285993A1 · Liu · 2020 [cited by examiner]
US 20200304545A1 · Rao · 2020 [cited by examiner]
US 20200364614A1 · Trott · 2020 [cited by examiner]
US 20210089910A1 · Zheng · 2021 [cited by examiner]
US 20210174246A1 · Triplet · 2021 [cited by examiner]
US 20210240776A1 · Jawagal · 2021 [cited by examiner]
US 20220130546A1 · Wei · 2022 [cited by examiner]
US 20220366245A1 · Guez · 2022 [cited by examiner]
US 20240046070A1 · Zaheer · 2024 [cited by examiner]
US 20240086703A1 · Ritter · 2024 [cited by examiner]
Li, Siyuan et al. Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards. 2019, Conference on Neural Information Processing Systems. <https://proceedings.neurips.cc/paper_files/paper/2019/file/81e74d… [cited by examiner]
Huang, Shengyi et al. Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games. Oct. 5, 2020, Cornell University. <https://doi.org/10.48550/arXiv.2010.03956> (Year: 2020). [cited by examiner]
Zhu, Xiaohui et al. Meta-Reward Model Based on Trajectory Data with k-Nearest Neighbors Method. Sep. 28, 2020, IEEE. <https://doi.org/10.1109/IJCNN48605.2020.9207388> (Year: 2020). [cited by examiner]
Memarian, Farzan et al. Self-Supervised Online Reward Shaping in Sparse-Reward Environments. Dec. 16, 2021, IEEE. <https://doi.org/10.1109/IROS51168.2021.9636020> (Year: 2021). [cited by examiner]
Abadi et al., “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv:1603.04467, Mar. 2016, 19 pages. [cited by applicant]
Abbeel et al., “Apprenticeship learning via inverse reinforcement learning,” Proceedings of the twenty-first international conference on Machine learning, Jul. 2004, 8 pages. [cited by applicant]
Abolafia et al., “Neural program synthesis with priority queue training,” arXiv preprint arXiv:1801.03526, Jan. 2018, 16 pages. [cited by applicant]
Agarwal et al., “Learning to generalize from sparse and underspecified rewards,” International Conference on Machine Learning, May 2019, 11 pages. [cited by applicant]
Amodei et al., “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565, Jun. 2016, 29pages. [cited by applicant]
Antol et al., “Vqa: Visual question answering,” Proceedings of the IEEE international conference on computer vision, 2015, 2425-2433. [cited by applicant]
Artzi et al., “Weakly supervised learning of semantic parsers for mapping instructions to actions,” Transactions of the Association for Computational Linguistics, Mar. 2013, 1:49-62. [cited by applicant]
Bahdanau et al., “Learning to understand goal specifications by modelling reward,” arXiv preprint arXiv:1806.01946, Jun. 2018, 19 pages. [cited by applicant]
Berant et al., “Explaining queries over web tables to non-experts,” 2019 IEEE 35th International Conference on Data Engineering (ICDE), Apr. 2019, 1570-1573. [cited by applicant]
Berant et al., “Semantic parsing on freebase from question-answer pairs,” Proceedings of the 2013 conference on empirical methods in natural language processing, Oct. 2013, 1533-1544. [cited by applicant]
Bosselut et al., “Discourse-aware neural rewards for coherent text generation,” arXiv preprint arXiv:1805.03766, May 2018, 12 pages. [cited by applicant]
Chaplot et al., “Gated-attention architectures for task-oriented language grounding,” Proceedings of the AAAI Conference on Artificial Intelligence, Apr. 2018, 32:1. [cited by applicant]
Chen et al., “Learning to interpret natural language navigation instructions from observations,” Proceedings of the AAAI Conference on Artificial Intelligence, Aug. 2011, 25:1. [cited by applicant]
Chevalier-Boisvert et al., “Babyai: A platform to study the sample efficiency of grounded language learning,” arXiv preprint arXiv:1810.08272, Oct. 2018, 19 pages. [cited by applicant]
Cho et al., “Adversarial tableqa: Attention supervision for question answering on tables,” Asian Conference on Machine Learning, Nov. 2018, 391-406. [cited by applicant]
Christiano et al., “Deep reinforcement learning from human preferences,” arXiv preprint arXiv:1706.03741, Jun. 2017, 17 pages. [cited by applicant]
Cobbe et al., “Quantifying generalization in reinforcement learning,” International Conference on Machine Learning, May 2019, 1282-1289. [cited by applicant]
Dayan et al., “Using expectation-maximization for reinforcement learning,” Neural Computation, Feb. 1997, 9(2):271-8. [cited by applicant]
Desautels et al., “Parallelizing exploration-exploitation tradeoffs in gaussian process bandit optimization,” Journal of Machine Learning Research, 2014, 15:3873-923. [cited by applicant]
Duan et al., “R1 $∧ 2$: Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779, Nov. 2016, 14 pages. [cited by applicant]
Finn et al., “Model-agnostic meta-learning for fast adaptation of deep networks,” International Conference on Machine Learning, Jul. 2017, 1126-1135. [cited by applicant]
Fu et al., “From language to goals: Inverse reinforcement learning for vision-based instruction following,” arXiv preprint arXiv:1902.07742, Feb. 2019, 14 pages. [cited by applicant]
Gleave et al., “Multi-task maximum entropy inverse reinforcement learning,” arXiv preprint arXiv:1805.08882, May 2018, 9 pages. [cited by applicant]
Golovin et al., “Google vizier: A service for black-box optimization,” Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2017, 10 pages. [cited by applicant]
Gupta et al., “Meta-reinforcement learning of structured exploration strategies,” arXiv preprint arXiv:1802.07245, Feb. 2018, 11 pages. [cited by applicant]
Guu et al., “From language to programs: Bridging reinforcement learning and maximum marginal likelihood,” arXiv preprint arXiv:1704.07926, Apr. 2017, 12 pages. [cited by applicant]
Haug et al., “Neural multi-step reasoning for question answering on semi-structured tables,” European conference on information retrieval, Mar. 2018, 611-617. [cited by applicant]
Hermann et al., “Grounded language learning in a simulated 3d world,” arXiv preprint arXiv:1706.06551, Jun. 2017, 22 pages. [cited by applicant]
Ho et al., “Generative adversarial imitation learning,” arXiv preprint arXiv:1606.03476, Jun. 2016, 14 pages. [cited by applicant]
Huang et al., “Natural language to structured query generation via meta-learning,” arXiv preprint arXiv:1803.02400, Mar. 2018, 9 pages. [cited by applicant]
Ibarz et al., “Reward learning from human preferences and demonstrations in atari,” arXiv preprint arXiv:1811.06521, Nov. 2018, 20 pages. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, Dec. 2014, 15 pages. [cited by applicant]
Krishnamurthy et al., “Neural semantic parsing with type constraints for semi-structured tables,” Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Sep. 2017, 1516-1526. [cited by applicant]
Langley et al., “Crafting papers on machine learning,” 17th International Conference on Machine Learning, Jun. 2000, 6 pages. [cited by applicant]
Leike et al., “Scalable agent alignment via reward modeling: a research direction,” arXiv preprint arXiv:1811.07871, Nov. 2018, 30 pages. [cited by applicant]
Liang et al., “Memory augmented policy optimization for program synthesis and semantic parsing,” arXiv preprint arXiv:1807.02322, Jul. 2018, pages. [cited by applicant]
Liang et al., “Neural symbolic machines: Learning semantic parsers on freebase with weak supervision,” ar Xiv preprint arXiv:1611.00020, Oct. 2016, 12 pages. [cited by applicant]
Liu et al., “Self-supervised generalisation with meta auxiliary learning,” arXiv preprint arXiv:1901.08933, Jan. 2019, 13 pages. [cited by applicant]
Long et al., “Simpler context-dependent logical forms via model projections,” arXiv preprint arXiv:1606.05378, Jun. 2016, 10 pages. [cited by applicant]
Maclaurin et al., “Gradient-based hyperparameter optimization through reversible learning,” International conference on machine learning, Jun. 2015, 2113-2122. [cited by applicant]
Močkus et al., “On Bayesian methods for seeking the extremum,” Optimization techniques IFIP technical conference, 1975, 400-404. [cited by applicant]
Mudrakarta et al., “It was the training data pruning too,” arXiv preprint arXiv:1803.04579, Mar. 2018, 3 pages. [cited by applicant]
Nachum et al., “Improving policy gradient by exploring under-appreciated rewards,” arXiv preprint arXiv:1611.09321, Nov. 2016, 17 pages. [cited by applicant]
Nagabandi et al., “Deep online learning via meta-learning: Continual adaptation for model-based rl,” arXiv preprint arXiv:1812.07671, Dec. 2018, 15 pages. [cited by applicant]
Neelakantan et al., “Learning a natural language interface with neural programmer,” arXiv preprint arXiv:1611.08945, Nov. 2016, 13 pages. [cited by applicant]
Nichol et al., “On first-order meta-learning algorithms,” arXiv preprint arXiv:1803.02999, Mar. 2018, 15 pages. [cited by applicant]
Norouzi et al., “Reward augmented maximum likelihood for neural structured prediction,” Advances In Neural Information Processing Systems, 2016, 29:1723-31. [cited by applicant]
Pasupat et al., “Compositional semantic parsing on semi-structured tables,” arXiv preprint arXiv:1508.00305, Aug. 2015, 11 pages. [cited by applicant]
Pasupat et al., “Inferring logical forms from denotations,” arXiv preprint arXiv:1606.06900, Jun. 2016, 10 pages. [cited by applicant]
Rasmussen et al., “Gaussian processes in machine learning,” Advanced lectures on machine learning, 2004, 2 pages. [cited by applicant]
Ren et al., “Learning to reweight examples for robust deep learning,” International Conference on Machine Learning, Jul. 2018, 4334-4343. [cited by applicant]
Shi et al., “Toward diverse text generation with inverse reinforcement learning,” arXiv preprint arXiv:1804.11258, Apr. 2018, 8 pages. [cited by applicant]
Snoek et al., “Practical bayesian optimization of machine learning algorithms,” arXiv preprint arXiv:1206.2944, Jun. 2012, 12 pages. [cited by applicant]
Wang et al., “Learning to reinforcement learn,” arXiv preprint arXiv:1611.05763, Nov. 2016, 17 pages. [cited by applicant]
Wang et al., “No metrics are perfect: Adversarial reward learning for visual storytelling,” arXiv preprint arXiv:1804.09160, Apr. 2018, 15 pages. [cited by applicant]
Williams et al., “Function optimization using connectionist reinforcement learning algorithms,” Connection Science, Jan. 1991, 3(3):241-68. [cited by applicant]
Williams et al., “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, May 1992, 8(3-4):229-56. [cited by applicant]
Winograd et al., “Procedures as a representation for data in a computer program for understanding natural language,” Massachusetts Inst of Tech Cambridge Project Mac, Feb. 1971, 472 pages. [cited by applicant]
Winograd et al., “Understanding natural language,” Cognitive Psychology, Jan. 1972, 3(1):1-91. [cited by applicant]
Wu et al., “Learning to teach with dynamic loss functions,” arXiv preprint arXiv:1810.12081, Oct. 2018, 15 pages. [cited by applicant]
Wu et al., “Sequence Prediction with Unlabeled Data by Reward Function Learning,” IJCAI, Jan. 2017, 3098-3104. [cited by applicant]
Xie et al., “Few-shot goal inference for visuomotor learning and planning,” Conference on Robot Learning, Oct. 2018, 40-52. [cited by applicant]
Xu et al., “Learning a prior over intent via meta-inverse reinforcement learning,” International Conference on Machine Learning, May 2019, 6952-6962. [cited by applicant]
Xu et al., “Learning to explore via meta-policy gradient,” International Conference on Machine Learning, Jul. 2018, 5463-5472. [cited by applicant]
Xu et al., “Meta-gradient reinforcement learning,” arXiv preprint arXiv:1805.09801, May 2018, 17 pages. [cited by applicant]
Yih et al., “The value of semantic parse labeling for knowledge base question answering,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Aug. 2016, 201-206. [cited by applicant]
Zelle et al., “Learning to parse database queries using inductive logic programming,” Proceedings of the national conference on artificial intelligence, Aug. 1996, 1050-1055. [cited by applicant]
Zhang et al., “A study on overfitting in deep reinforcement learning,” arXiv preprint arXiv:1804.06893, Apr. 2018, 25 pages. [cited by applicant]
Zhang et al., “Macro grammars and holistic triggering for efficient semantic parsing,” arXiv preprint arXiv:1707.07806, Jul. 2017, 14 pages. [cited by applicant]
Zhao et al., “Metric-optimized example weights,” International Conference on Machine Learning, May 2019, 7533-7542. [cited by applicant]
Zheng et al., “On learning intrinsic rewards for policy gradient methods,” arXiv preprint arXiv:1804.06459, Apr. 2018, 15 pages. [cited by applicant]
Zhong et al., “Seq2sql: Generating structured queries from natural language using reinforcement learning,” arXiv preprint arXiv:1709.00103, Aug. 2017, 12 pages. [cited by applicant]
Ziebart et al., “Maximum entropy inverse reinforcement learning,” Aaai, Jul. 2008, 8:1433-1438. [cited by applicant]
Zou et al., “Reward shaping via meta-learning,” arXiv preprint arXiv:1901.09330, Jan. 2019, 12 pages. [cited by applicant]