Computer code generation from task descriptions using neural networks

1 . A method performed by one or more computers, the method comprising:

receiving description data describing a computer programming task, the description data comprising a plurality of text tokens;

generating a plurality of initial candidate computer programs, the generating comprising sampling a plurality of output sequences from a set of one or more generative neural networks, wherein each generative neural network in the set is deployed as a plurality of instances on a respective set of one or more devices, and wherein each instance of each generative neural network in the set is configured to, in parallel with each other instance of each generative neural network, receive an input sequence that includes the plurality of text tokens in the description data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specifies an initial candidate computer program for performing the computer programming task described by the description data, wherein each initial candidate computer program is specified by a respective one of the plurality of output sequences;

filtering the plurality of initial candidate computer programs to generate a plurality of candidate computer programs, wherein the filtering comprises:

compiling each of the initial candidate computer programs; and

removing, from the plurality of initial candidate computer programs, any initial candidate computer program that fails to compile;

clustering the plurality of candidate computer programs to generate a plurality of clusters that each comprise a respective plurality of candidate computer programs;

ranking the plurality of clusters using at least a respective cardinality of each of the clusters;

selecting, for inclusion in a set of one or more clusters that includes less than all of the plurality of clusters, one or more highest-ranked clusters according to the ranking;

for each cluster in the set of one or more of the clusters:

processing each of the respective plurality of candidate computer programs in the cluster using a correctness estimation neural network to generate a correctness score for the candidate computer program that estimates a likelihood that the candidate computer program accurately performs the computer programming task; and

selecting a representative computer program for the cluster using the correctness scores for the respective plurality of candidate computer programs in the cluster; and

selecting one or more of the representative computer programs for the clusters as synthesized computer programs for performing the computer programming task.

2 . The method of claim 1 , further comprising:

obtaining a first set of one or more inputs for the computer programming task and, for each input in the first set, one or more expected outputs, wherein the filtering further comprises:

discarding any initial candidate computer program that does not generate, for each input in the first set, an output that matches any of the one or more expected outputs for the input.

3 . The method of claim 1 , wherein clustering the plurality of candidate computer programs comprises:

obtaining a test set of test inputs for the computer programming task;

for each candidate computer program and for each test input in the test set:

executing the candidate computer program on the test input to generate an output; and

clustering the plurality of candidate computer programs based at least in part on the outputs generated by executing the candidate computer programs on the test inputs.

4 . The method of claim 3 , wherein obtaining the test set of test inputs for the computer programming task comprises:

processing a natural language description of inputs for the computer programming task using a test input generation neural network to generate as output one or more test inputs for the computer programming task.

5 . The method of claim 4 , wherein processing the natural language description using a test input generation neural network comprises:

processing a description sequence generated from the natural language description using a language model neural network to generate a specification of the inputs to the computer programming task in a compact domain specific language; and

generating the one or more test inputs from the specification of the inputs to the computer programming task.

6 . The method of claim 1 , wherein each generative neural network has been fine-tuned on a respective first set of code generation training data in accordance with a respective set of hyperparameters.

7 . The method of claim 6 , wherein each generative neural network has been further fine-tuned on a respective second set of code generation training data.

8 . The method of claim 1 , wherein ranking the plurality of clusters using at least a respective cardinality of each of the clusters comprises:

ranking the plurality of clusters based on, for each of the clusters, a respective cardinality of the cluster and a respective measure of central tendency of correctness scores generated by the correctness estimation neural network for the candidate computer programs in the cluster.

9 . The method of claim 1 , further comprising:

receiving a new input for the computer programming task; and

executing one or more of the synthesized computer programs on the new input to generate an output for the computer programming task for the new input.

10 . The method of claim 1 , wherein sampling a plurality of output sequences from a set of one or more generative neural networks comprises sampling each output sequence in accordance with a respective sampling temperature and wherein two or more output sequences are sampled with respective different sampling temperatures.

11 . The method of claim 1 , wherein one or more of the generative neural networks are causally-masked decoder-only Transformer neural networks.

12 . The method of claim 1 , wherein one or more of the generative neural networks in the set include:

an encoder neural network configured to process the input sequence to generate an encoded representation of the input sequence; and

a decoder neural network configured to process the encoded representation to generate the output sequence.

13 . The method of claim 12 , wherein the encoder neural network is a Transformer encoder that applies self-attention over the input sequence and the decoder neural network is an auto-regressive Transformer decoder that applies cross-attention into the encoded representation.

14 . The method of claim 1 , wherein sampling a plurality of output sequences from a set of one or more generative neural networks comprises repeatedly performing operations comprising:

generating a current input sequence from the description data; and

sampling one or more output sequences by processing the current input sequence using one of the generative neural networks in the set.

15 . The method of claim 14 , wherein each input sequence comprises the plurality of text tokens from the description data and one or more tokens identifying a programming language, and wherein generating a current input sequence from the description data comprises:

selecting a programming language; and

including one or more tokens identifying the selected programming language in the current input sequence.

16 . The method of claim 14 , wherein generating a current input sequence from the description data comprises:

including one or more tokens indicating that the output sequence generated by the generative neural network will specify a correct solution for the computer programming task.

17 . The method of claim 14 , wherein each input sequence comprises the plurality of text tokens from the description data and zero or more tags that each comprise one or more tokens and that identify an algorithm that will be implemented by the output sequence generated by the generative neural network, and wherein generating a current input sequence from the description data comprises:

sampling one or more tags from a distribution over a set of tags; and

including the sampled tags in the current output sequence.

18 . The method of claim 14 , wherein each input sequence comprises the plurality of text tokens from the description data and one or more tokens that specify a difficulty rating for the computer programming task, and wherein generating a current input sequence from the description data comprises:

selecting a difficulty rating for the computer programming task; and

including one or more tokens specifying the selected difficulty rating in the current output sequence.

19 . The method of claim 18 , wherein selecting a difficulty rating for the computer programming task comprises:

selecting a fixed difficulty rating that is the same for all computer programming tasks after training of the generative neural networks.

20 . The method of claim 18 , wherein selecting a difficulty rating for the computer programming task comprises:

sampling a difficulty rating from a distribution over a set of difficulty ratings.

21 . The method of claim 14 , wherein each input sequence comprises the plurality of text tokens from the description data and a sequence of tokens that represent a natural language description of how to solve the computer programming task.

22 . The method of claim 21 , further comprising:

generating, from the description data, the sequence of tokens that represent the natural language description using an editorial generation neural network.

23 . The method of claim 1 , wherein each generative neural network has been pre-trained on a language model training objective on a language model training data set and fine-tuned on a code generation objective on a respective first set of code generation training data that includes a plurality of training examples that each include (i) a training input sequence representing a computer programming task and (ii) a training output sequence that represents a computer program.

24 . The method of claim 23 , wherein the plurality of training examples includes a first subset of training examples for which the training output sequence in the training example is a correct solution to the computer programming task represented by the training input sequence in the training example and a second set of training examples for which the training output sequence in the training example is not a correct solution to the computer programming task represented by the training input sequence in the training example.

25 . The method of claim 1 , wherein processing each of the respective plurality of candidate computer programs in the cluster using a correctness estimation neural network to generate a correctness score for the candidate computer program that estimates a likelihood that the candidate computer program accurately performs the computer programming task comprises, for each candidate computer program, processing a correctness input sequence generated from the candidate computer program using the correctness estimation neural network to generate a correctness score for the candidate computer program.

26 . The method of claim 25 , wherein the correctness input sequence is generated from the candidate computer program and at least a subset of the description data.

27 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving description data describing a computer programming task, the description data comprising a plurality of text tokens;

generating a plurality of initial candidate computer programs, the generating comprising sampling a plurality of output sequences from a set of one or more generative neural networks, wherein each generative neural network in the set is deployed as a plurality of instances on a respective set of one or more devices, and wherein each instance of each generative neural network in the set is configured to, in parallel with each other instance of each generative neural network, receive an input sequence that includes the plurality of text tokens in the description data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specifies an initial candidate computer program for performing the computer programming task described by the description data, wherein each initial candidate computer program is specified by a respective one of the plurality of output sequences;

filtering the plurality of initial candidate computer programs to generate a plurality of candidate computer programs, wherein the filtering comprises:

compiling each of the initial candidate computer programs; and

removing, from the plurality of initial candidate computer programs, any initial candidate computer program that fails to compile;

clustering the plurality of candidate computer programs to generate a plurality of clusters that each comprise a respective plurality of candidate computer programs;

ranking the plurality of clusters using at least a respective cardinality of each of the clusters;

selecting, for inclusion in a set of one or more clusters that includes less than all of the plurality of clusters, one or more highest-ranked clusters according to the ranking;

for each cluster in the set of one or more of the clusters:

processing each of the respective plurality of candidate computer programs in the cluster using a correctness estimation neural network to generate a correctness score for the candidate computer program that estimates a likelihood that the candidate computer program accurately performs the computer programming task; and

selecting a representative computer program for the cluster using the correctness scores for the respective plurality of candidate computer programs in the cluster; and

selecting one or more of the representative computer programs for the clusters as synthesized computer programs for performing the computer programming task.

28 . The system of claim 27 , wherein the operations further comprise:

obtaining a first set of one or more inputs for the computer programming task and, for each input in the first set, one or more expected outputs, wherein the filtering further comprises:

discarding any initial candidate computer program that does not generate, for each input in the first set, an output that matches any of the one or more expected outputs for the input.

29 . The system of claim 27 , wherein clustering the plurality of candidate computer programs comprises:

obtaining a test set of test inputs for the computer programming task;

for each candidate computer program and for each test input in the test set:

executing the candidate computer program on the test input to generate an output; and

clustering the plurality of candidate computer programs based at least in part on the outputs generated by executing the candidate computer programs on the test inputs.

30 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving description data describing a computer programming task, the description data comprising a plurality of text tokens;

generating a plurality of initial candidate computer programs, the generating comprising sampling a plurality of output sequences from a set of one or more generative neural networks, wherein each generative neural network in the set is deployed as a plurality of instances on a respective set of one or more devices, and wherein each instance of each generative neural network in the set is configured to, in parallel with each other instance of each generative neural network, receive an input sequence that includes the plurality of text tokens in the description data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specifies an initial candidate computer program for performing the computer programming task described by the description data, wherein each initial candidate computer program is specified by a respective one of the plurality of output sequences;

filtering the plurality of initial candidate computer programs to generate a plurality of candidate computer programs, wherein the filtering comprises:

compiling each of the initial candidate computer programs; and

removing, from the plurality of initial candidate computer programs, any initial candidate computer program that fails to compile;

clustering the plurality of candidate computer programs to generate a plurality of clusters that each comprise a respective plurality of candidate computer programs;

ranking the plurality of clusters using at least a respective cardinality of each of the clusters;

selecting, for inclusion in a set of one or more clusters that includes less than all of the plurality of clusters, one or more highest-ranked clusters according to the ranking;

for each cluster in the set of one or more of the clusters:

processing each of the respective plurality of candidate computer programs in the cluster using a correctness estimation neural network to generate a correctness score for the candidate computer program that estimates a likelihood that the candidate computer program accurately performs the computer programming task; and

selecting a representative computer program for the cluster using the correctness scores for the respective plurality of candidate computer programs in the cluster; and

selecting one or more of the representative computer programs for the clusters as synthesized computer programs for performing the computer programming task.