Conversational unit test generation using large language model
A large language model, trained on source code and natural language text generates a unit test for a change to a file in a pull request of a code repository. An ordered sequence of prompts is created and each is applied serially to the large language model to perform an individual task that leads to the generation of the unit test. The unit test may be added to an existing file or generated as a newly-created file. Each prompt includes the data from a previously-issued prompt of the ordered sequence in order for the model to retain contextual knowledge learned previously. The model generates the unit test as update commands when the unit test is added to an existing file.
1 . A system comprising:
one or more processors; and
a memory that stores one or more programs that are configured to be executed by the one or more processors, the one or more programs comprising instructions to perform acts that:
access a large language model trained on source code and natural language to generate source and/or natural language given a prompt;
detect a change to a file from a pull request of a code repository, wherein the change is not associated with a unit test;
construct a first prompt for the large language model to generate a location in the code repository to incorporate a first unit test to test the change to the file from the pull request of the code repository, wherein the first prompt comprises a structure of the code repository;
cause the large language model to generate a location to incorporate the first unit test given the first prompt;
when the large language model indicates an existing file in the code repository as the location to incorporate the first unit test:
construct a second prompt for the large language model to generate edits to the existing file that add the first unit test to the existing file, wherein the second prompt comprises contents of the existing file in the code repository;
cause the large language model to generate the edits given the second prompt; and
incorporate the edits into the existing file; and
when the large language model indicates a new file in the code repository as the location to incorporate the first unit test:
obtain a comparison test file from the code repository;
cause the large language model to generate a new test file given the comparison test file and the change to the file from the pull request; and
place the new test file into the code repository.
2 . The system of claim 1 , wherein the structure of the code repository includes a directory of the changed file, a parent directory of the directory of the changed file, a subdirectory of the directory of the changed file, and/or a directory having a pathname that includes a test keyword.
3 . The system of claim 2 , wherein the one or more programs comprise instructions to perform acts that:
prioritize contents of the first prompt and/or the second prompt to reduce the contents of the first prompt and/or second prompt to fit into a context window size of the large language model.
4 . The system of claim 1 , wherein the one or more programs comprise instructions to perform acts that:
check syntax of the source code of the existing file with the edits; and
if a syntax error is found, correct the syntax error.
5 . The system of claim 1 , wherein the second prompt includes a format for the edits to the existing file, wherein the format includes update commands that insert lines before or after a line of the existing file and replace a span of lines of the existing file with replaced content.
6 . The system of claim 1 , wherein detect a change to a file from a pull request of a code repository comprises instructions to perform act that: determine whether the change to the file is testworthy.
7 . The system of claim 6 , wherein the one or more programs include comprise instructions to perform acts that:
cause the large language model to determine whether the change to the file is testworthy given the change to the file.
8 . The system of claim 1 , wherein the large language model is a neural transformer model with attention.
9 . A computer-implemented method, comprising:
accessing a large language model trained on source code and natural language to generate source code and/or natural language given a prompt;
receiving a change to a file from a pull request of a code repository, wherein the change is not associated with a unit test;
creating a first prompt for the large language model to predict a location to incorporate the unit test for the change to the file from the pull request in the code repository, wherein the first prompt includes a structure of the code repository and the changed file;
causing the large language model to generate the location given the first prompt;
receiving from the large language model the location of the unit test;
when the large language model indicates a new file as the location for the unit test:
creating a second prompt for the large language model to generate the unit test, wherein the second prompt includes the first prompt and a comparison test file from the code repository;
causing the large language model to generate the unit test given the change to the file from the pull request;
receiving from the large language model the unit test given the second prompt; and
placing the new file into the code repository; and
when the large language model indicates an existing file in the code repository as the location for the unit test:
causing the large language model to generate the edits to the existing file given contents of the existing file in the code repository; and
incorporating the edits into the existing file.
10 . The computer-implemented method of claim 9 , further comprising:
prior to creating the first prompt, determining whether the change to the file is testworthy.
11 . The computer-implemented method of claim 10 , further comprising:
causing the large language model to determine whether the change to the file is testworthy given the change to the file.
12 . The computer-implemented method of claim 11 , further comprising:
reducing size of the fourth prompt to fit within a context window size of the large language model by prioritizing lines of the selected comparison file to include in the fourth prompt.
13 . The computer-implemented method of claim 9 , further comprising:
checking for syntax errors in the unit test generated by the large language model; and
upon detecting a syntax error, correcting the syntax error.
14 . The computer-implemented method of claim 9 , further comprising:
reducing size of the first prompt to fit within a context window size of the large language model by prioritizing directories of the structure of the code repository to include into the first prompt.
15 . The computer-implemented method of claim 9 , wherein the large language model is a neural transformer model with attention.
16 . A computer-implemented method, comprising:
accessing a large language model trained on source code and natural language to generate a unit test for a change to a file from a pull request of a code repository;
creating an ordered sequence of prompts for the large language model to generate the unit test, wherein a prompt includes an instruction to perform a task and an answer format, wherein a subsequent prompt in the ordered sequence includes a previous prompt in the ordered sequence, wherein a first prompt of the ordered sequence includes a first instruction for the large language model to determine whether the change to the file from the pull request is testworthy, wherein a second prompt of the ordered sequence includes a second instruction for the large language model to determine a location to incorporate the unit test in the code repository, wherein a third prompt of the ordered sequence includes a third instruction for the large language model to generate contents of the unit test relative to the determined location;
applying each prompt of the ordered sequence serially to the large language model;
obtaining, from the large language model, a response to each prompt; and
upon receiving a response to the third prompt, storing the unit test in the location in the code repository.
17 . The computer-implemented method of claim 16 , wherein the second prompt includes a structure of the code repository.
18 . The computer-implemented method of claim 16 ,
wherein the determined location is an existing file in the code repository, and
wherein the third prompt includes contents of the existing file.
19 . The computer-implemented method of claim 18 ,
wherein the third prompt includes an answer format, the answer format includes update commands that represent edits to the existing file that add the unit test.
20 . The computer-implemented method of claim 19 , further comprising:
applying the edits to the existing file;
checking for syntax correctness of the existing file having the edits; and
correcting for syntax errors in the existing file having the edits.