Computer code generation from task descriptions using neural networks
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating computer code using neural networks. One of the methods includes receiving description data describing a computer programming task; receiving a first set of inputs for the computer programming task; generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks; for each candidate computer program in a subset of the candidate computer programs and for each input in the first set: executing the candidate computer program on the input to generate an output; and selecting, from the candidate computer programs, one or more computer programs as synthesized computer programs for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs.
1 . A method performed by one or more computers, the method comprising:
receiving description data describing a computer programming task, the description data comprising a plurality of text tokens;
receiving a first set of inputs for the computer programming task;
generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, wherein each generative neural network in the set is deployed as a plurality of instances on a respective set of one or more devices, and wherein each instance of each generative neural network in the set is configured to, in parallel with each other instance of each generative neural network, receive an input sequence that includes the plurality of text tokens in the description data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specifies a candidate computer program for performing the computer programming task described by the description data;
for each candidate computer program in a subset of the candidate computer programs and for each input in the first set:
executing the candidate computer program on the input on the respective set of one or more devices on which the instance that generated the candidate computer program is deployed to generate an output; and
selecting, from the candidate computer programs, one or more computer programs as synthesized computer programs for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs, comprising:
clustering the candidate computer programs in the subset into a plurality of clusters based on the outputs generated by executing the candidate computer programs on the inputs in the first set; and
selecting, as a synthesized computer program, one or more respective candidate computer programs from each of one or more of the clusters.
2 . The method of claim 1 , further comprising:
receiving a new input for the computer programming task; and
executing one or more of the synthesized computer programs on the new input to generate an output for the computer programming task for the new input.
3 . The method of claim 1 , further comprising:
receiving, for each input in the first set, a respective expected output generated by performing the computer programming task on the input, wherein the selecting comprises:
for each candidate computer program in the subset and for each input in the first set:
determining whether the output generated by executing the candidate computer program on the input matches the respective expected output for the input; and
selecting, as an initial set of computer programs, only candidate computer programs that, for each input in the first set, generated an output that matched the respective expected output for the input when executed on the input;
wherein selecting, from the candidate computer programs, one or more computer programs as synthesized computer programs for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs comprises selecting, as a synthesized computer program, one or more respective computer programs from the initial set of computer programs.
4 . The method of claim 1 , wherein the subset of candidate computer programs includes all of the plurality of candidate computer programs.
5 . The method of claim 1 , wherein selecting, as a synthesized computer program, one or more respective computer programs from each of one or more of the clusters comprises:
selecting, as a synthesized computer program, one or more respective computer programs from each of a predetermined number of largest clusters of the plurality of clusters.
6 . The method of claim 1 , wherein obtaining the first set of inputs for the computer programming task comprises:
processing one or more description sequences that each include the plurality of text tokens from the description data using a test input generation neural network that is configured to process each description sequence to generate as output one or more test inputs for the computer programming task described by the description sequence.
7 . The method of claim 1 , wherein the subset of candidate computer programs includes only candidate computer programs that were not filtered out from the plurality of candidate computer programs based on, for each candidate computer program and for each input in a second set of inputs, whether the output generated by executing the candidate computer program on the input matched a respective expected output for the input.
8 . The method of claim 1 , wherein the set of one or more generative neural networks includes only a single generative neural network.
9 . The method of claim 1 , wherein the set of one or more generative neural networks includes a plurality of generative neural networks, wherein each generative neural network in the set has a different combination of values for a set of one or more properties than each other neural network in the set, and wherein the properties include one or more of (i) training data used to train the generative neural network, (ii) a number of parameters of the generative neural network, or (iii) a sampling temperature for sampling output sequences from the generative neural network.
10 . The method of claim 1 , wherein one or more of the generative neural networks in the set include:
an encoder neural network configured to process the input sequence to generate an encoded representation of the input sequence; and
a decoder neural network configured to process the encoded representation to generate the output sequence.
11 . The method of claim 10 , wherein the encoder neural network is a Transformer encoder that applies self-attention over the input sequence and the decoder neural network is an auto-regressive Transformer decoder that applies cross-attention into the encoded representation.
12 . The method of claim 1 , wherein generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks comprises repeatedly performing operations comprising:
generating a current input sequence from the description data; and
sampling one or more output sequences by processing the current input sequence using one of the generative neural networks in the set.
13 . The method of claim 12 , wherein each input sequence comprises the plurality of text tokens from the description data and one or more tokens identifying a programming language, and wherein generating a current input sequence from the description data comprises:
sampling a programming language from a distribution over a set of programming languages; and
including one or more tokens identifying the programming language in the current input sequence.
14 . The method of claim 12 , wherein each input sequence comprises the plurality of text tokens from the description data and one or more tokens indicating whether the output sequence generated by the generative neural network should specify a correct or incorrect solution for the computer programming task, and wherein generating a current input sequence from the description data comprises:
including one or more tokens indicating that the output sequence generated by the generative neural network should specify a correct solution for the computer programming task.
15 . The method of claim 12 , wherein each input sequence comprises the plurality of text tokens from the description data and zero or more tags that each comprise one or more tokens and that identify an algorithm that should be implemented by the output sequence generated by the generative neural network, and wherein generating a current input sequence from the description data comprises:
sampling one or more tags from a distribution over a set of tags; and
including the sampled tags in the current output sequence.
16 . The method of claim 12 , wherein each input sequence comprises the plurality of text tokens from the description data and one or more tokens that specify a difficulty rating for the computer programming task, and wherein generating a current input sequence from the description data comprises:
selecting a difficulty rating for the computer programming task; and
including one or more tokens specifying the selected difficulty rating in the current output sequence.
17 . The method of claim 16 , wherein selecting a difficulty rating for the computer programming task comprises:
selecting a fixed difficulty rating that is the same for all computer programming tasks after training of the generative neural networks.
18 . The method of claim 16 , wherein selecting a difficulty rating for the computer programming task comprises:
sampling a difficulty rating from a distribution over a set of difficulty ratings.
19 . The method of claim 1 , wherein each generative neural network has been pre-trained on a language modeling objective on a first set of training data and fine-tuned on a code generation objective on a second set of training data that includes a plurality of training examples that each include (i) a training input sequence representing a computer programming task and (ii) a training output sequence that represents a computer program.
20 . The method of claim 19 , wherein the first set of training data comprises a set of sequences that each represent a segment of computer code.
21 . The method of claim 19 , wherein the plurality of training examples includes a first subset of training examples for which the training output sequence in the training example is a correct solution to the computer programming task represented by the training input sequence in the training example and a second set of training examples for which the training output sequence in the training example is not a correct solution to the computer programming task represented by the training input sequence in the training example.
22 . The method of claim 19 , wherein the code generation objective is GOLD with tempering.
23 . The method of claim 1 , wherein the text tokens and the computer code tokens are selected from a same vocabulary of tokens.
24 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving description data describing a computer programming task, the description data comprising a plurality of text tokens;
receiving a first set of inputs for the computer programming task;
generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, wherein each generative neural network in the set is deployed as a plurality of instances on a respective set of one or more devices, and wherein each instance of each generative neural network in the set is configured to, in parallel with each other instance of each generative neural network, receive an input sequence that includes the plurality of text tokens in the description data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specifies a candidate computer program for performing the computer programming task described by the description data;
for each candidate computer program in a subset of the candidate computer programs and for each input in the first set:
executing the candidate computer program on the input on the respective set of one or more devices on which the instance that generated the candidate computer program is deployed to generate an output; and
selecting, from the candidate computer programs, one or more computer programs as synthesized computer programs for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs, comprising:
clustering the candidate computer programs in the subset into a plurality of clusters based on the outputs generated by executing the candidate computer programs on the inputs in the first set; and
selecting, as a synthesized computer program, one or more respective candidate computer programs from each of one or more of the clusters.
25 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving description data describing a computer programming task, the description data comprising a plurality of text tokens;
receiving a first set of inputs for the computer programming task;
generating a plurality of candidate computer programs by sampling a plurality of output sequences from a set of one or more generative neural networks, wherein each generative neural network in the set is deployed as a plurality of instances on a respective set of one or more devices, and wherein each instance of each generative neural network in the set is configured to, in parallel with each other instance of each generative neural network, receive an input sequence that includes the plurality of text tokens in the description data and to process the input sequence to generate an output sequence comprising a plurality of computer code tokens that specifies a candidate computer program for performing the computer programming task described by the description data;
for each candidate computer program in a subset of the candidate computer programs and for each input in the first set:
executing the candidate computer program on the input on the respective set of one or more devices on which the instance that generated the candidate computer program is deployed to generate an output; and
selecting, from the candidate computer programs, one or more computer programs as synthesized computer programs for performing the computer programming task based at least in part on the outputs generated by executing the candidate computer programs in the subset on the inputs in the first set of inputs, comprising:
clustering the candidate computer programs in the subset into a plurality of clusters based on the outputs generated by executing the candidate computer programs on the inputs in the first set; and
selecting, as a synthesized computer program, one or more respective candidate computer programs from each of one or more of the clusters.