Code generation with reinforcement learning

1. A system comprising:

a processor and a memory;

wherein the memory includes executable instructions that when executed by the processor perform actions that:

access a first deep learning model configured to predict whether a non-terminal symbol exists to be expanded given a partial-code state that represents a partially-formed code snippet;

access a second deep learning model configured to predict an expansion of a non-terminal symbol predicted by the first deep learning model, wherein the expansion replaces the predicted non-terminal symbol in accordance with a production rule of a grammar of a programming language of the partially-formed source code snippet;

obtain the partially-formed source code snippet; and

execute the first deep learning model and the second deep learning model to generate a code sketch to complete the partially-formed source code snippet, wherein the code sketch includes a source code snippet with at least one hole.

2. The system of claim 1 , wherein the memory includes executable instructions that when executed by the processor perform actions that:

output the code sketch in a program containing the partially-formed source code snippet.

3. The system of claim 1 , wherein the memory includes executable instructions that when executed by the processor perform actions that:

parse the partially-formed source code snippet into a target partial-code state, the target partial-code state including at least one non-terminal symbol; and

apply the target partial-code state to the first deep learning model to determine the top-M predicted non-terminal symbols to expand from the target partial-code state.

4. The system of claim 3 , wherein the memory includes executable instructions that when executed by the processor perform actions that:

for each of the top-M predicted non-terminal symbols, execute the second deep learning model to predict top-N expansions of each predicted non-terminal; and

generate additional partial-code states for each of the M*N expansions.

5. The system of claim 4 , wherein the memory includes executable instructions that when executed by the processor perform actions that:

accumulate an aggregated probability for each of the M*N expansions, the aggregated probability based on a probability of each expansion and a probability of each predicted non-terminal symbol in the partial-code state that is expanded.

6. The system of claim 4 , wherein the memory includes executable instructions that when executed by the processor perform actions that:

select, as code sketches, top-k partial-code states based on a highest aggregated probability; and

replace each non-terminal symbol in the code sketches with a hole.

7. The system of claim 1 , wherein the code sketch generates source code to complete a line of source code, a method body, a class, an expression, or a consecutive set of lines of source code.

8. The system of claim 1 , wherein the first deep learning model is a multilayer perceptron model trained through reinforcement learning.

9. The system of claim 1 , wherein the second deep learning model is an encoder-decoder neural transformer model with attention trained through reinforcement learning.

10. A computer-implemented method, comprising:

accessing a non-terminal expansion model that predicts whether or not to expand a non-terminal symbol given a code sketch representing a partially-formed source code snippet;

obtaining a first training dataset including a plurality of partial-code states;

training a non-terminal selector model, through reinforcement learning, to predict a non-terminal symbol to expand using the first training dataset; and

fine-tuning the non-terminal expansion model and the non-terminal selector model jointly, using the first training dataset to maximize a reward based on a predicted code sketch and a true terminal sequence.

11. The method of claim 10 , further comprising:

creating a supervised training dataset including a plurality of triplets, a triplet including a partial-code state, an index of a position of a non-terminal symbol and a true non-terminal expansion; and

training the non-terminal expansion model using the supervised training dataset.

12. The method of claim 11 , wherein the reward is based on an evaluation metric applied to the predicted code sketch and the true terminal sequence.

13. The method of claim 12 , wherein the non-terminal selector model is a multilayer perceptron.

14. The method of claim 13 , wherein training the non-terminal selector model further comprises:

updating weights of the multilayer perceptron based on a value of the reward.

15. The method of claim 14 , wherein the non-terminal expansion model is an encoder-decoder neural transformer model with attention.

16. The method of claim 15 , wherein fine-tuning the non-terminal expansion model and the non-terminal selector model jointly further comprises:

updating weights of the encoder-decoder neural transformer model with attention based on outcome of the reward.

17. A device, comprising:

a processor and a memory;

wherein the processor is configured to perform acts that:

access a pre-trained non-terminal expansion model configured to predict an expansion of a non-terminal symbol of a partial-code state, wherein the partial-code state represents a partially-formed code snippet, the partial-code state having at least one non-terminal symbol;

execute the pre-trained non-terminal expansion model to predict a code sketch given a target partial-code state; and

train a non-terminal selector model to learn through reinforcement learning to predict a non-terminal symbol to expand in the target partial-code sketch, wherein the reinforcement learning includes application of a reward function based on a comparison of the predicted code sketch with a true terminal sequence.

18. The device of claim 17 , wherein the reward function optimizes a discrete non-differentiable metric.

19. The device of claim 17 , wherein the pre-trained non-terminal expansion model is an encoder-decoder neural transformer model with attention trained on a supervised dataset.

20. The device of claim 17 , wherein the processor is configured to perform acts that:

fine-tune the non-terminal selector model and the non-terminal expansion model jointly through reinforcement learning using a plurality of partial-code states and corresponding true terminal sequences.