Neural network systems for source code generation and ranking

1. A computer-implemented method for generating and ranking code solutions for performing a task, the method comprising:

receiving, by one or more processors, input data comprising: (i) a task description describing a task, (ii) a code generation prompt that instructs at least one trained code generation neural network to generate source code for performing the task, and (iii) a test case generation prompt that instructs the at least one trained code generation neural network to generate a set of test cases for testing the source code generated by the at least one trained code generation neural network, wherein the at least one trained code generation neural network comprises a plurality of artificial neurons arranged into a plurality of neural network layers;

processing the input data using the at least one trained code generation neural network to generate (i) a plurality of code solutions, each of the plurality of code solutions comprising a respective set of candidate source code for performing the task, and (ii) a plurality of test cases for testing the plurality of code solutions, wherein each of the plurality of test cases comprises a test input and an expected output for the test input;

for each of the plurality of code solutions:

executing, by the one or more processors, the set of candidate source code in the code solution on the test inputs of the plurality of test cases to generate a plurality of execution outputs for the test inputs;

clustering, by the one or more processors, the plurality of code solutions into a plurality of clusters based on the execution outputs of the plurality of code solutions;

computing, by the one or more processors, an interaction matrix that specifies functional overlap between the plurality of clusters;

determining, for each of the plurality of clusters, a score based the interaction matrix;

ranking the plurality of clusters based on the scores of the plurality of clusters; and

returning a generated code solution from a cluster of the plurality of clusters that has a highest score.

2. The computer-implemented method of claim 1 , wherein clustering, by the one or more processors, the plurality of code solutions into the plurality of clusters based on the execution outputs of the plurality of code solutions comprises:

grouping code solutions that have identical execution outputs into a same cluster.

3. The computer-implemented method of claim 1 , wherein determining, for each of the plurality of clusters, a score based the interaction matrix comprises:

multiplying the interaction matrix by a validation score vector to obtain a ranking score vector that comprises a score for each of the plurality of clusters.

4. The computer-implemented method of claim 3 , wherein the validation score vector represents, for each of the plurality of clusters, a feature of the cluster.

5. The computer-implemented method of claim 4 , wherein the feature is a number of code solutions in the cluster, or

wherein the feature is a number of test cases that the code solutions in the cluster have passed.

6. The computer-implemented method of claim 1 , wherein the interaction matrix is denoted as Iϵ K×K , where K is a number of clusters in the plurality of clusters, wherein each element in the interaction matrix is computed as follows:

I

i

⁢

j

=

1

M

⁢

∑

k

=

1

M

⁢

δ

⁡

(

o

i

⁢

k

=

o

j

⁢

k

)

,

where I ij represents functional overlap between cluster C i and cluster C j , o ik and o jk are execution outputs of cluster C i and cluster C j respectively on a k th test input, δ is an indicator function that returns 1 if the execution outputs o ik are identical to the execution outputs o jk and returns 0 otherwise, and M is the number of test cases in the set of test cases.

7. The computer-implemented method of claim 1 , further comprising:

receiving a new input;

selecting a code solution from a cluster having the highest score among the plurality of clusters; and

executing the source code in the selected code solution on the new input to perform the task.

8. A neural network system for generating and ranking source code, the neural network system comprising one or more computers and one or more non-transitory computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving input data comprising (i) a task description describing a task, (ii) a code generation prompt that instructs at least one trained code generation neural network to generate source code for performing the task, and (iii) a test case generation prompt that instructs the code generation neural network to generate a set of test cases for testing the source code generated by the at least one trained code generation neural network, wherein the at least one trained code generation neural network comprises a plurality of artificial neurons arranged into a plurality of neural network layers;

processing the input data using the at least one trained code generation neural network to generate (i) a plurality of code solutions, each of the plurality of code solutions comprising a respective set of candidate source code for performing the task, and (ii) a plurality of test cases for testing the plurality of code solutions, wherein each of the plurality of test cases comprises a test input and an expected output for the test input;

for each of the plurality of code solutions, executing the set of candidate source code in the code solution on the test inputs of the plurality of test cases to generate a plurality of execution outputs for the test inputs;

clustering the plurality of code solutions into a plurality of clusters based on the execution outputs of the plurality of code solutions;

computing an interaction matrix that specifies functional overlap between the plurality of clusters;

determining, for each of the plurality of clusters, a score based the interaction matrix;

ranking the plurality of clusters based on the scores of the plurality of clusters; and

returning a generated code solution from a cluster of the plurality of clusters that has a highest score.

9. The neural network system of claim 8 , wherein clustering the plurality of code solutions into the plurality of clusters based on the execution outputs of the plurality of code solutions comprises:

grouping code solutions that have identical execution outputs into a same cluster.

10. The neural network system of claim 8 , wherein determining, for each of the plurality of clusters, a score based the interaction matrix comprises:

multiplying the interaction matrix by a validation score vector to obtain a ranking score vector that comprises a score for each of the plurality of clusters.

11. The neural network system of claim 10 , wherein the validation score vector represents, for each of the plurality of clusters, a feature of the cluster.

12. The neural network system of claim 11 , wherein the feature is a number of code solutions in the cluster.

13. The neural network system of claim 11 , wherein the feature is a number of test cases that the code solutions in the cluster passed.

14. The neural network system of claim 8 , wherein the interaction matrix is denoted as Iϵ K×K , where K is a number of clusters in the plurality of clusters, wherein each element in the interaction matrix is computed as follows:

I

i

⁢

j

=

1

M

⁢

∑

k

=

1

M

⁢

δ

⁡

(

o

i

⁢

k

=

o

j

⁢

k

)

,

where I ij represents functional overlap between cluster C i and cluster C j , o ik and o jk are execution outputs of cluster C i and cluster C j respectively on a kfth test input, δ is an indicator function that returns 1 if the execution outputs o ik are identical to the execution outputs o jk and returns 0 otherwise, and M is the number of test cases in the set of test cases.

15. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating and ranking code solutions, the operations comprising:

receiving input data, wherein the input data comprises (i) a task description describing a task, (ii) a code generation prompt that instructs at least one trained code generation neural network to generate source code for performing the task, and (iii) a test case generation prompt that instructs the at least one trained code generation neural network to generate a set of test cases for testing the source code generated by the trained code generation neural network, wherein the at least one trained code generation neural network comprises a plurality of artificial neurons arranged into a plurality of neural network layers;

processing the input data using the at least one trained code generation neural network to generate (i) a plurality of code solutions, each of the plurality of code solutions comprising a respective set of candidate source code for performing the task, and (ii) a plurality of test cases for testing the plurality of code solutions, wherein each of the plurality of test cases comprises a test input and an expected output for the test input;

for each of the plurality of code solutions:

executing the set of candidate source code in the code solution on the test inputs of the plurality of test cases to generate a plurality of execution outputs for the test inputs;

clustering the plurality of code solutions into a plurality of clusters based on the execution outputs of the plurality of code solutions;

computing an interaction matrix that specifies functional overlap between the plurality of clusters;

determining, for each of the plurality of clusters, a score based the interaction matrix;

ranking the plurality of clusters based on the scores of the plurality of clusters; and

returning a generated code solution from a cluster of the plurality of clusters that has a highest score.

16. The one or more non-transitory computer storage media of claim 15 , wherein the operations for clustering, by the one or more processors, the plurality of code solutions into the plurality of clusters based on the execution outputs of the plurality of code solutions comprises:

grouping code solutions that have identical execution outputs into a same cluster.

17. The one or more non-transitory computer storage media of claim 15 , wherein the operations for determining, for each of the plurality of clusters, a score based the interaction matrix comprises:

multiplying the interaction matrix by a validation score vector to obtain a ranking score vector that comprises a score for each of the plurality of clusters.

18. The one or more non-transitory computer storage media of claim 17 , wherein the validation score vector represents, for each of the plurality of clusters, a feature of the cluster.

19. The one or more non-transitory computer storage media of claim 18 , wherein the feature is a number of code solutions in the cluster, or wherein the feature is a number of test cases that the code solutions in the cluster passed.

20. The one or more non-transitory computer storage media of claim 15 , wherein the interaction matrix is denoted as ϵ K×K , where K is the number of clusters in the plurality of clusters, wherein each element in the interaction matrix is computed as follows:

I

i

⁢

j

=

1

M

⁢

∑

k

=

1

M

⁢

δ

⁡

(

o

i

⁢

k

=

o

j

⁢

k

)

,

where I ij represents functional overlap between cluster C i and cluster C j , o ik and o jk are execution outputs of cluster C i and cluster C j respectively on the k th test input, δ is an indicator function that returns 1 if the execution outputs o ik are identical to the execution outputs o jk and returns 0 otherwise, and M is the number of test cases in the set of test cases.